# Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition
## 4 Experiments
### Effectiveness  
This thesis uses precision (P), recall (R), and the F1 score (F1) as metrics tested on two benchmark MNER datasets. For an entity, it is judged to be correctly predicted only if both its boundaries and categories are correctly predicted. Precision is the proportion of correct entities among the predicted entities, and recall is the proportion of correctly predicted entities among all entities in the sample. A high precision rate indicates that the model predicted a high percentage of correct entities but did not necessarily cover more entities. A high recall indicates that the model predicted most of the entities in the data; it covered more entities but did not necessarily predict a high proportion of correct entities. The F1 value is a weighted average of the precision and recall, taking into account both the precision of the prediction and the number of correct entities covered. Table 2 displays the performance of four text-based models and eight multimodal models on Twitter-2015, while Table 3 presents their performance on Twitter-2017. Below is a detailed analysis.  
Firstly, comparing all text-based NER methods, it is evident from the data that methods based on BERT outperform others, indicating the potential advantage of optimizing pretrained models through transfer learning rather than starting from scratch. Additionally, approaches combining BERT with the CRF surpass pure BERT strategies, suggesting the crucial role of the CRF in capturing label constraints and thereby achieving more accurate label predictions.  
Next, we compare MNER methods with competitive text-based models, such as GVATT-HBiLSTM-CRF and HBiLSTM-CRF. The majority of multimodal strategies notably outperform their corresponding text-only models, further indicating the contribution of image information to named entity recognition in text. Furthermore, compared to the latest MNER method, DebiasCL, our GMF method, incorporating both the proposed visual guidance module and consistency matching module, enhances image information aligned with the text.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Modality**} & \multirow{2}{*}{**Methods**} & \multicolumn{6}{c}{**Twitter-2015**} & \multirow{2}{*}{**Overall**} \\  & & & **Single Type (F1)** & & & & **Overall** \\  & & **PER** & **LOC** & **ORG** & **MISC** & **P** & **R** & **F1** \\ \hline \multirow{4}{*}{Text} & BiLSTM-CRF & 76.77 & 72.56 & 41.33 & 26.8 & 68.14 & 61.09 & 64.42 \\  & HBiLSTM-CRF & 82.34 & 76.83 & 51.59 & 32.52 & 70.32 & 68.05 & 69.17 \\  & BERT & 84.72 & 79.91 & 58.26 & 38.81 & 68.3 & 74.61 & 71.32 \\  & BERT-CRF & 84.74 & 80.51 & 60.27 & 37.29 & 69.22 & 74.59 & 71.81 \\ \hline \multirow{4}{*}{Text+image} & GVATT-HBiLSTM-CRF & 82.66 & 77.21 & 55.06 & 35.25 & 73.96 & 67.9 & 70.8 \\  & GVATT-BERT-CRF & 84.43 & 80.87 & 59.02 & 38.14 & 69.15 & 74.46 & 71.7 \\  & BPCL & 85.6 & 81.77 & 63.81 & 40.3 & 74.02 & 75.07 & 74.54 \\  & UMT-BERT-CRF & 85.24 & 81.58 & 63.03 & 39.45 & 71.67 & 75.23 & 73.41 \\  & MAF & 84.67 & 81.18 & 63.35 & 41.82 & 71.86 & 75.1 & 73.42 \\  & UMGF & 84.26 & **83.17** & 62.45 & 42.42 & 74.49 & 75.21 & 74.85 \\  & DebiasCL & 85.97 & 81.84 & **64.02** & 43.38 & 74.45 & 76.13 & 75.28 \\  & **GMF (ours)** & **87.02** & 82.51 & 63.25 & **45.2** & **75.47** & **76.15** & **75.81** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experimental results on the Twitter-2015 dataset.  
Finally, on Twitter-2015 and Twitter-2017, our model shows overall F1 improvements of 0.53% and 0.27%, respectively. This suggests that the introduced modules can assist the model in better integrating text and image representations.