# Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition
###### Abstract
[11] proposes a new multilevel semantic alignment approach that captures coarse- to fine-grained interactions between images and language and directly utilizes learned visual features to capture the relationship between images and text more comprehensively, avoiding the complexity of using external tools and datasets. In addition, ref. [12] had a unique idea [12; 13; 14] to utilize the external matching between different (text, image) pair relationships and designed an R-GCN model to model these relationships, using the external matching relationships between different (text, image) pairs within the dataset to mitigate image noise in the MNER task while modeling both cross-modal and intra-modal relationships. Ref. [13] proposed the SD-NER model, which models the minimum distance matrix between entities and is easily transferable to other tasks. Ref. [14] proposed a multimodal Chinese named entity recognition (USAF) model using acoustic features, which unifies textual and acoustic features through a unique positional embedding and fuses the features of both features using a multi-head attention mechanism. Although existing methods are a great improvement [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14], they have three main shortcomings. Firstly, existing methods assume that each piece of text and its corresponding  
Figure 1: An example of multimodal tweets. In this tweet, “[PERSON]” is the name of the dog. image are matched and that the image can help recognize named entities in the text, but not all texts are matched with their corresponding images. As shown in Figure 2, ref. [7] incorrectly predicts \"Aquamarine\" as ORG due to the influence of the house, indicating that the existing model has difficulties in filtering the noise introduced by the mismatched text-image pairs. According to [15], about 34.1% of the text content in Twitter-2015 has an imperfect match with images. Secondly, existing work on images usually relies only on the Residual Network (ResNet) [16] or Mask-RCNN [17] and other image encoders to extract visual features without enhancement or filtering and feed them directly into cross-modal interaction mechanisms along with text. Finally, most existing approaches fail to construct a consistent representation to bridge the semantic gap between the two modalities. As shown in Figure 3, ideally, the words \"Danielle\" and \"Melissa\" in the text should show a high degree of similarity with the region associated with the \"rabbits\" in the image and low similarity with other regions in the image. However, due to the inconsistent representation between the text and the image, the similarity between \"Danielle\" and \"Melissa\" in the text and the \"rabbits\" in the image may be lower than the similarity between \"Danielle\" and \"Melissa\" in the text and the \"rabbits\" in the image in the calculation of the similarity score. The similarity between \"Danielle\" and \"Melissa\" in the text and the \"rabbits\" in the image may be lower than the similarity with other regions in the image. To address these shortcomings, we propose a visual clue guidance and consistency matching framework (GMF). Firstly, we design a visual clue guidance module (VCG), which collects image features through the Resnet50 image encoder. On this basis, we propose a cross-scale attention (CSA) module, which combines low-level, high-resolution image information with high-level, strong semantic image information to enhance visual features. Further, feature fusion is performed by a structured mapping function \(F(\theta)\), and, finally, the above vectors are processed using a gating mechanism for path decision-making. Secondly, we utilize Contrastive Language-Image Pretraining (CLIP) [18] based on the idea of multimodal contrast learning, and we design the consistency matching module (CM), defining contrast loss to bring matched image-text pairs closer together and push them away, enhancing the learning of the multimodal potential semantic space, and making the representation of two modalities more consistent.