# A Dual-Enhanced Hierarchical Alignment Framework for Multimodal Named Entity Recognition
###### Abstract  
Multimodal amed entity recognition (MNER) is a natural language-processing technique that integrates text and visual modalities to detect and segment entity boundaries and their types from unstructured multimodal data. Although existing methods alleviate semantic deficiencies by optimizing image and text feature extraction and fusion, a fundamental challenge remains due to the lack of fine-grained alignment caused by cross-modal semantic deviations and image noise interference. To address these issues, this paper proposes a dual-enhanced hierarchical alignment (DEHA) framework that achieves dual semantic and spatial enhancement via global-local cooperative alignment optimization. The proposed framework incorporates a dual enhancement strategy comprising Semantic-Augmented Global Contrast (SAGC) and Multi-scale Spatial Local Contrast (MS-SLC), which reinforce the alignment of image and text modalities at the global sample level and local feature level, respectively, thereby reducing image noise. Additionally, a cross-modal feature fusion and vision-constrained CRF prediction layer is designed to achieve adaptive aggregation of global and local features. Experimental results on the Twitter-2015 and Twitter-2017 datasets yield F1 scores of 77.42% and 88.79%, outperforming baseline models. These results demonstrate that the global-local complementary mechanism effectively balances alignment precision and noise robustness, thereby enhancing entity recognition accuracy in social media and advancing multimodal semantic understanding. m +
Footnote †: journal: applied sciences  
0000-0001-6000-0000-0000-0000-0000-0000-0000-0000-0000-00000-00000-00000-00000-00000-00000-00000-00000-00000-00000-00000-000000-00000-00000-000000-00000-000  
rapidly developing and is gradually becoming a core technology in downstream tasks such as knowledge graph enhancement and multimodal dialogue systems. Multimodal named entity recognition (MNER) leverages cross-modal complementary information by integrating textual and visual features to enhance recognition accuracy [3], demonstrating superior performance and remarkable effectiveness in social media scenarios where noise interference is severe. However, existing MNER methods still face three key challenges: Firstly, semantic deviation between modalities leads to insufficient cross-modal alignment. [PERSON] [4] found that relying solely on cosine similarity for feature matching overlooks entity-level semantic consistency. As shown in Figure 1a, when the text mentions \"apple\", the existing method UMT [5] fails to effectively distinguish between the fruit (which is not labeled as an entity) and the company (ORG) due to the lack of fine-grained semantic supervision, resulting in entity type misclassification. Secondly, the issue of visual noise interference is significant. Experiments by [PERSON] et al. [6] indicate that irrelevant image regions may mislead the model into incorrectly classifying textual entities. For instance, when the text describes the event \"[PERSON] is giving a speech\" (Figure 1b), the Washington Monument in the background (visual noise) causes the model [7] to misclassify the person's name (PER) as a location (LOC). Thirdly, existing contrastive learning methods primarily focus on global sample alignment (e.g., MoCo [8]) and lack supervision for local fine-grained semantic units, leading to ambiguous entity boundaries [9]. As illustrated in Figure 1c, CLIP-ner [10] struggles with long-tail entities such as \"Springfield\" (LOC); due to its single-scale feature matching mechanism, it fails to capture the hierarchical differences in multimodal semantics, misclassifying it as an organization (ORG). To tackle the challenges outlined above, this paper introduces the dual-enhanced hierarchical alignment (DEHA) framework, which leverages a global-local collaborative optimization strategy to achieve dual enhancement of semantics and spatial alignment.