Boosting Textural NER with Synthetic Image and Instructive Alignment (2024)

Abstract: Named entity recognition (NER) is a pivotal task reliant on textual data, often impeding the disambiguation of entities due to the absence of context. To tackle this challenge, conventional methods often incorporate images crawled from the internet as auxiliary information. However, the images often lack sufficient entities or would introduce noise. Even with high-quality images, it is still challenging to achieve fine-grained alignment with texts. We introduce a novel method named InstructNER to address these issues. Leveraging the rich real-world knowledge and image synthesis capabilities of a large pre-trained stable diffusion model, InstructNER transforms the textonly NER into a multimodal NER (MNER) task. A selection process automatically identifies the best synthetic image by comparing fine-grained similarities with internet-crawled images through a visual bag-of-words strategy. Note, during the image synthesis, a cross-attention matrix between synthetic images and raw text emerges, which inspires a soft attention guidance alignment (AGA) mechanism. AGA optimizes the MNER task and concurrently facilitates instructive alignment in MNER. Experiments on prominent MNER datasets show that our method surpasses all textonly baselines, improving F1-score by 1.4% to 2.3%. Remarkably, even when compared to fully multimodal baselines, our approach maintains competitive. Furthermore, we opensource a comprehensive synthetic image dataset and the code to supplement existing raw dataset. The code and datasets are available in https: //github.com/Heyest/InstructNER.

heading: 1 Introduction
text: Named entity recognition (NER) is a fundamental information extraction task that identifies named
*Corresponding author. †Now working in Meta.
entities in sentences and classifies them into predefined categories(Li et al., 2020). Text-based NER methods in practice mainly depend on textual information, which poses challenges in recognizing short or ambiguous sentences (Zhang et al., 2018). In comparison with text-based NER, multimodal NER (MNER) leverages images as supplementary information to boost the task’s robustness (Xu et al., 2022; Jia et al., 2022; Wang et al., 2023a). In Figure 1 (a), the word ICRC in the sentence I love ICRC is ambiguous, hindering the classification based solely on the text. An additional image clarifies that ICRC refers to the International Committee of the Red Cross, labeled as ORG.
Despite the promising results of MNER methods (Chen et al., 2021), they still struggle with acquiring large-scale and high-quality text-image paired corpus. Common MNER datasets, Twitter-2015 (Zhang et al., 2018) and Twitter-2017 (Lu et al., 2018), consist of 8,257 and 4,819 image-text pairs, respectively. This represents a mere 26% and 16% of the volume found in the frequently employed textual NER dataset CoNLL03 (Tjong Kim Sang and De Meulder, 2003). One potential solution involves crawling supplementary images by internet search engines. Whereas, unlike raw texts, the internetcrawled images often fail to cover enough entities,
which might further bring unexpected noise. As exemplified in Figure 1 (b), the two images are retrieved from Google based on Messi is playing football at FCB home stadium Camp Nou, where the left image only contains Messi while the right one only includes Camp Nou.
Another critical challenge to the success of the MNER task lies in the accurate alignment between text tokens and image regions. Existing techniques for the text-image alignment can be summarized into two groups: explicit alignment and implicit alignment. Explicit alignment extracts visual object regions, then maps them to corresponding textual tokens (Wu et al., 2020; Zheng et al., 2020; Jia et al., 2022), but errors can propagate from initial inaccurate region extraction (Yang et al., 2019a). Consider the left image in Figure 1 (c), only the object regions of Messi and football are extracted, missing the FCB clothes logo and Camp Nou stadium in the background. In contrast, implicit alignment alleviates this issue by employing an attention mechanism to learn the alignment weights adaptively (Zhou et al., 2022; Xu et al., 2022; Wang et al., 2023a). Despite the token-to-image alignment achieved by these approaches, the attention may not be sufficiently concentrated. Regarding the right image of Figure 1 (c), the attention heatmap illustrates Messi is located, yet it noticeably allocates irrelevant attention to surrounding areas.
In response to the aforementioned challenges, we propose InstructNER, a novel approach that harnesses the rich real-world knowledge and the image synthesis capabilities of stable diffusion models (Rombach et al., 2022) to provide supplemental information, thereby transforming the textual NER task into MNER task. Specifically, we first feed the raw text into the pre-trained stable diffusion (SD) model to generate large-scale synthetic images. Then, to alleviate the variability of image quality and select the best one, we employ an off-the-shelf visual bag-of-words (BoW) method (Gidaris et al., 2020) to measure the finegrained similarity between the internet-crawled images and the synthetic images, ultimately selecting the most similar one. Particularly, the similarity metric of images considers both the coverage and accuracy of entities. As depicted in Figure 1 (d) (left), the selected image accurately encompasses all entities, including Messi, the FCB logo, and the Camp Nou stadium. Furthermore, the image synthesis process also produces a cross-attention matrix between the synthetic images and the raw
text as a byproduct, inspiring our soft attention guidance alignment (AGA) mechanism for MNER model training. In addition to optimizing the original MNER task, our objective also aims to minimize the Kullback-Leibler (KL) divergence between the MNER model’s cross-attention matrix and the aforementioned byproduct matrix. Figure 1 (d) shows the attention of the AGA mechanism paid to token Messi, displaying a higher concentration.
We conduct experiments on three representative MNER datasets, i.e., Twitter-2015, Twitter-2017, and WikiDiverse, while excluding images with only textual corpus. Experimental results demonstrate the superiority of our method over all text-only baselines, with absolute F1-score improvements of 1.4%, 2.3% and 2.1%. Moreover, our method still achieves competitive results even when compared to fully multimodal baselines. To sum up, the contributions of this paper are three-fold:
• We are the first to leverage the artificial intelligence generated content (AIGC) ability of stable diffusion model to switch textual NER into MNER with synthetic images.
• We propose a comprehensive framework InstructNER with a novel text-to-image mechanism AGA, which instructs the cross-attention being learned in a soft manner.
• Experimental results compared with both textonly and fully MNER baselines verify the effectiveness of our method. Moreover, we release the large-scale and high-quality synthetic images to supplement the raw datasets.

heading: 2 Method
text: NER aims to categorize named entities in a sentence S = (s1, s2, . . . , sn) consisting of n tokens, and often adopts the BIO tagging schema (Sang and Veenstra, 1999). The output Y = (y1, y2, . . . , yn) consists of n labels, where yi ∈ L and L = {B, I,O} represents the predefined label set. The MNER task receives an image as additional input, then identifies named entities similar to NER.
Figure 2 provides an overview of the comprehensive architecture of InstructNER. In Stage #1, InstructNER leverages the pre-trained stable diffusion model to generate images, and selects the optimal synthetic image covering sufficient and accurate entities. In Stage #2, InstructNER utilizes the soft attention guidance alignment (AGA) mechanism to fuse the raw text and synthetic image. In
Stage #3, InstructNER adopts a multi-task training object to cocurrently optimize the loss of entity classification and instructive alignment.

heading: 2.1 Image Synthesis and Selection
text: For the sentence S = (s1, s2, . . . , sn), we first generate l synthetic images I = {I1, I2, . . . , Il} using the pre-trained SD model (Rombach et al., 2022), and then select the best one Iτ ∈ I considering the entity coverage and accuracy.

heading: 2.1.1 Image Selection with Entity Coverage
text: For each synthetic image Ii ∈ I , we derive its cross attention matrix Msyni from pre-trainted SD model, where Msyni = (mi,1,mi,2, . . . ,mi,n), and mi,j ∈ Rdp represents the attention score vector between token sj and image Ii (decomposed into dp pixels). Notably, the entity tokens (e.g., Messi and Camp Nou) could receive higher attention scores, while function tokens (e.g., is and at) obtain lower attentions. For each token sj , its importance wj is measured by summing up the average attention score µi,j , which corresponds to all the image pixels:
µi,j = 1
dp dp∑ k=1 mi,j [k], wj = l∑ i=1 µi,j (1)
We use a predefined threshold θ to select the entity tokens. In particular, wj ≥ θ indicates that token sj is the desired entity token. We can then filter out the images which exhibit low attention scores with the entity tokens. To this end, we take this strategy to rank the quality of the synthetic image set I , and keep α best images among them, denoted as I ′, where I ′ ⊂ I and α is a hyper-parameter.

heading: 2.1.2 Image Selection with Entity Accuracy
text: The synthetic image set I ′ excel in entity coverage, but may include some unrealistic and inaccurate contents. Conversely, internet-crawled images often exhibit high factual but lack sufficient entity coverage. Combing the advantage of these two types of images, we employ internet-crawled images to filter synthetic images and select the optimal image Iτ from I ′.
Specifically, we crawl m images C = {C1, C2, . . . , Cm} from a search engine. For a specific image Ci ∈ I ′ ∪ C, we utilize the pretrained RotNet encoder (Gidaris et al., 2018) to obtain visual features vi = (v1i ,v 2 i , . . . ,v r i ), which consist of r regions, where vri ∈ Rc represents a c-dimensional vector. To measure the fine-grained similarity between C and I ′, an off-the-shelf visual bag-of-words (BoW) model (Gidaris et al., 2020) is employed. K-Means algorithm is first adopted in clustering all the region features {vi} ∣∣|I′∪C| i=1 within the image set I ′ ∪ C into k-cluster, in order to calculate the image similarity with vocabulary consisting of k vectors. Formally, the vocabulary F = (f1, f2, . . . , fk) can be represented as:
(f1, f2, . . . , fk) = KMeans ( v1,v2, . . . ,v|I′∪C| ) (2) where fk ∈ Rc denote a specific vector. Then, we quantize each region of the image Ci with the most similar (the minimum squared Euclidean distance) feature in the vocabulary, and compute the visual bag-of-words representation bow(Ci).
bow(Ci) = { bow(Cji ) } ∣∣∣k j=1
(3)
bow(Cji ) = 1 (ED(fj ,vi) < η) (4)
where 1[·] is the indicator operator, ED(·) denotes the minimum squared Euclidean distance, η is a threshold manually defined. Following the steps above, we compute the BoW representations for all images in I ′∪C, and select the most similar image in I ′ comparing with C as Iτ . Following the steps above, we compute the BoW representations for all images in I ′∪C, and select the most similar image in I ′ comparing with C as Iτ .
Iτ = arg min Ii∈I′ m∑ j=1 ED (bow(Ii), bow(Cj)) (5)
where ED(·) denotes the minimum squared Euclidean distance.

heading: 2.2 Attention Guided Text-Image Alignment
text: After obtaining the optimal synthetic image Iτ , we reframe the text-only NER into MNER, i.e., first performing text and image representation and second conducting instructive multimodal alignment.

heading: 2.2.1 Text and Image Encoding
text: For sentence S, BERT (Devlin et al., 2019) is employed to encode contextualized textual representations Ĥt ∈ Rn×d, where n and d denotes the sentence length and hidden dimension. Then, to further guide the interaction within textual modality, a self-attention transformer (Vaswani et al., 2017) is utilized. We first map the initial text representation Ĥt as query Qt, key Kt and value Vt by different linear projections, where Qt,Kt,Vt ∈ Rn×d, then use attention mechanism to obtain the interacted text representations Ht ∈ Rn×d.
Mt = softmax ( QtK T t√
dk
) ,Ht = LN(Ĥt+MtVt)
(6) where Mt ∈ Rn×n, LN denotes the normalization function of transformer layer . Moreover, for the image Iτ , we resize it to 224 × 224 pixels and derive the visual output ResNet(Iτ ) = {hj |hj ∈ R2048, j = 1, 2, . . . , 49} from the last convolutional layer of pre-trained ResNet (He et al., 2016). The obtained output is divided into 49 regions, with each region as a 2048-dimensional vector hj . To align these visual representations with the textual counterparts, a linear mapping operation with parameter Wv ∈ Rd×2048 is applied, resulting in the final visual representation Hv ∈ R49×d.
Hv = Wv · ResNet(Iτ ) (7)

heading: 2.2.2 Instructive Text-Image Alignment
text: After obtaining the unimodal text representation Ht and image representation Hv respectively, we further utilize two cross-attention transformers (Yu et al., 2020) to facilitate interaction between modalities. As illustrated in Stage #2, we first takes Hv as query and Ht as key and value, learning the image-aware token representation Hcrst ∈ R49×d.
Hcrst = Cross-ATT(Hv,Ht,Ht) (8)
where Cross-ATT(·) denotes the cross-attention layers. Then, we take Ht as query and Hv as key and value, learning the token-aware visual representation:
Hcrsv = Cross-ATT(Ht,Hv,Hv) (9)
In order to guide the attention in the crossattention transformers to be more focused, our AGA mechanism utilize the byproduct of attention matrix Msynτ between each token and image Iτ generated by SD to guide the soft attention alignment. In the above two cross-attention transformers, two attention matrixes can be obtained similar to Mt in Equation 6. We average the two attention matrix to obtain Mcrs = {mcrs1 ,mcrs2 , . . . ,mcrsn }, where mcrsi ∈ R49 represents the attention scores between word si and the 49 regions of image Iτ . However, Msynτ = {msyn1 ,m syn 2 , . . . ,m syn n }, where msyni ∈ Rdp represents the attention between word si and all dp image pixels. To facilitate comparison, we use an average pooling operation to convert msyni to the same dimension as m crs i . Finally, we calculate the KL divergence between the two attention matrices and use it as an auxiliary loss.
Lalign = n∑
i=1
DKL (q (m crs i ) ||p(m syn i )) (10)
where q(·) and p(·) refer to transforming mcrsi and mgeni into attention distributions through a softmax layer, and DKL represents the KL divergence.

heading: 2.3 Model Training and Inference
text: During training, the visual representation Hcrsv and textual representation Hcrst are first fed into a gated network (Xu et al., 2022) for fusion, and then connected with a CRF layer to computes the probability of the label, where the labeling loss is denoted as Lcrf . During training, our method consists of two learning tasks, the CRF negtative log-likelihood loss for MNER task and the auxiliary loss for the AGA mechanism. We train the two tasks jointly,
and the final loss function is defined as follows:
L = Lcrf + λLalign (11)
where λ is a hyper-parameter to control the contribution of the auxiliary loss.
During inference, we switch textual NER into MNER task by synthesizing image in Stage #1. The raw text and synthetic image are fed into Stage #2 for alignment and fusion. Then the obtained multimodal representations are input into CRF layer to predict the labels Y = (y1, y2, . . . , yn).

heading: 3 Experiments

heading: 3.1 Datasets and Settings
text: We perform experiments on three representative datasets, Twitter-2015, Twitter-2017 and WikiDiverse (Wang et al., 2022), by excluding images that solely consist of textual corpus. Details of the datasets and metrics are described in the appendix A. We use the pre-trained stable diffusion 2.0-v to synthesize images1. During the image selection stage, we set the threshold θ to 0.05, η to 0.2 and the visual vocabulary size k to 100. For each textual sample, we generate 5 synthetic images and compare the similarity with 5 internet-crawled images, where α = 3 denotes the number of selected images for the first filtering procedure. We set the maximum sentence length n to 128, the epochs to 25, the mini-batch size to 32, the hidden dimension d to 768, and the number of attention heads to 12. The Adam optimizer (Kingma and Ba, 2014) is used with a learning rate of 5e-5, a dropout rate of 0.9, and an auxiliary loss weight λ of 0.5.

heading: 3.2 Baselines

heading: 3.2.1 Text-Only Models
text: (1) BiLSTM-CRF (Huang et al., 2015) is a classic NER model stacking a bidirectional LSTM layer and a CRF layer. (2) BERT-CRF (Liu et al., 2020) employs BERT as the encoder and CRF as the decoder. (3) BERT+BS (Zhu and Li, 2022) proposes boundary smoothing as a regularization technique for span-based NER models. (4) MultiNER (Wang et al., 2023b) proposes a multi-task learning framework for MRC-based NER.

heading: 3.2.2 Text-Image Models
text: (1) UMT (Yu et al., 2020) presents a multimodal interaction module to generate expressive text-visual
1https://github.com/Stability-AI/stablediffusion
representations. (2) MT (Yu et al., 2020) is the variation of UMT with the ablation of auxiliary entity span detection. (3) RpBERT (Sun et al., 2021) proposes a novel text-image relation propagation based multimodal BERT model. (4) UMGF (Zhang et al., 2021a) proposes a unified graph fusion approach to obtain the text-image representation. (5) MAF (Xu et al., 2022) is a general matching and alignment framework for the MNER task. (6) CogVLM (Weihan Wang, 2023) is the latest multimodal large model, fine-tuned here with LoRA (Hu et al., 2022) to serve as the newest baseline for comparison.

heading: 3.3 Main Results
text: The main results are reported in Table 1, from which we draw the following conclusions.
First, MNER methods generally perform better than text-only NER methods. Comparing the multimodal SOTA model MAF with its unimodal counterpart MultiNER, the former achieves F1 gains of 0.63%, 1.54%, and 1.11% on three datasets, respectively. The observation validates our motivation that image information functions as auxiliaries to provide enriched context for text-based NER. However, the fine-tuned CogVLM exhibits poor performance, attributed to the inherent limitations of generative multimodal large models in handling natural languange understanding tasks (i.e. NER), for which comprehension based models like BERT are inherently more adept.
Second, solely relying on the textual corpus of three datasets, InstructNER achieves F1 scores of 74.12%, 86.28%, and 74.46%, surpassing all textonly models. Notably, our method outperforms the best text-only model, MultiNER, by 1.41%, 2.30%, and 2.13%, demonstrating the efficacy of synthetic images as a supplement for enhancing textual NER.
Third, comparing Twitter-2017 with Twitter2015 and WikiDiverse, there exists a discrepancy of 12.16% and 11.82% of InstructNER’s F1-scores. The discrepancy is attributed to the shorter sentence length and the chaos derived from more ungrammatical sentences in Twitter-2015, like the internet slang YOLO, man. In WikiDiverse, the presence of multiple topics and more entity categories also increase the complexity of identification.
Last, when compared to existing multimodal methods that rely on sophisticated annotated images in datasets, our method still achieves remarkable results. InstructNER yields a substantial enhancement by 0.78%, 0.76%, and 1.02% compared to the best MNER method MAF. It is noteworthy that
our approach achieves the highest enhancement of 1.02% on WikiDiverse. This can potentially be attributed to the fact that WikiDiverse is sourced from the news domain, encompassing more enriched and diverse topics. Our method is capable of synthesizing images that incorporate various topicspecific scenarios. Furthermore, WikiDiverse includes more diverse entity categories, such as APP and Film. Through the guidance of the AGA mechanism, entities of these categories are more readily aligned with corresponding regions in images.

heading: 3.4 Ablation Experiments
text: To access individual component efficacy, we conduct ablation experiments on the full InstructNER model and its variants. As depicted in Table 2, we observe that the two image selection strategies (EC and EA) and the AGA mechanism significantly contribute to the final results. The performance decline follows the order: InstructNER w/o AGA >InstructNER w/o EA >InstructNER w/o EC. Specifically, removing the EC module results in a decrease of 0.49%, 0.43%, and 0.41% in the F1 score, while the removal of the EA module leads to performance drops of 0.87%, 0.82%, and 0.64%. This highlights the relatively more substantial factor by the EA than EC in improving performance. The primary reason behind this observation might be that the entity nouns are consistently portrayed in synthetic images, yet the fidelity or accuracy of these entities might not invariably be upheld. Moreover, removing the AGA module has the most significant decline of 1.17%, 1.20%, and 1.04%.
This supports the critical importance of AGA technique in aligning text and images.

heading: 3.5 Analysis Experiments

heading: 3.5.1 Effect under Different Sample Numbers
text: To examine the robustness of our models under varying dataset sizes, we conduct experiments by randomly sampling 2000 to 500 instances from three datasets. As shown in Figure 3, the performance of InstructNER is compared against two baseline models, MultiNER and BERT-CRF. It can be observed that when the number of training samples decreases, the F1 scores of the two text-only baseline models both decline rapidly. However, the decline trend of InstructNER is relatively slow. This observation indicates that recognizing named entities based solely on textual context becomes increasingly challenging when the training sample size is limited. To address this point, incorporating images as supplementary information significantly enhances the robustness of NER.

heading: 3.5.2 Effect of Multi-Task Learning Losses
text: We examine and illustrate the influence of the auxiliary loss weight λ on the AGA mechanism in Figure 4, providing valuable insights into its impact. Our first observation reveals that as λ varies from 0.3 to 0.7, the F1 scores for both datasets exhibit an upward trend, reaching their peak around the value of 0.5. However, further increasing the value of λ leads to a decline in F1 scores. Essentially, a higher value of λ indicates a stronger emphasis on textimage alignment within the model. Conversely, as λ decreases, the model places greater importance on learning the entity classification. When the losses from both modules are approximately balanced at a value of 0.5, the model achieves its highest overall performance. This balance allows for effective integration of both text-image alignment and entity classification, resulting in superior results across the board.

heading: 3.5.3 Effect of Synthetic Image Datasets
text: To validate the synthetic image datasets constructed by our method, we substitute the images in raw datasets with our synthetic ones and evaluate the performance on several existing MNER models. As shown in Table 4, our synthetic images exhibit comparable results to original images across all three models. Considering MAF model, using the synthetic images only results in a marginal decrease of 0.22%, 0.13%, and 0.18% in performance. This underscores the effectiveness of our synthetic image dataset. Without any manual annotation, our synthetic image datasets have achieved results closely resembling the original images at an remarkably low cost. Furthermore, even when using slightly less effective synthetic images, the inclusion of the
AGA mechanism has allowed our method to surpass MAF with raw images. This demonstrates the robustness of our AGA mechanism.

heading: 3.5.4 Effect under Different Sentence Length
text: To validate that InstructNER provides more auxiliary information for text-only NER tasks under the challenging condition of short sentence lengths, we categorize samples in the test sets based on sentence length and separately record recognition results for the text-only methods and InstructNER in Table 4. For sentences shorter than 10 words, MultiNER achieves the lowest F1 score at 70.11%, 81.68% and 69.69% on three datasets, underscoring the limitation of relying solely on textual context, particularly in shorter sentences. However, in this scenario, InstructNER shows the highest improvement in F1 score compared to MultiNER, reaching 1.75%, 3.08% and 2.55%. This suggests that our synthetic images provide more supplemental information when the textual context is limited.

heading: 3.5.5 Effect of BOW Strategy
text: We employ the siamese network approaches as an alternative to the BOW strategy, verifying the effect of BOW for image selection with entity accuracy. Specifically, pre-trained models VGG (Simonyan and Zisserman, 2015) and Vision Transformer (ViT) (Dosovitskiy et al., 2021) are utilized to extract image features for similarity measurement. The selected synthetic images are then fed into our model to perform MNER. Results in Table 5 demonstrate thay the utilization of the BOW strategy in extraction region-level image features, coupled with the comparison of fine-grained similarities is overall superior to the siamese network
methods. Notably, the BOW strategy achieve score improvements of 0.51%, 0.13%, 0.29% compared to ViT encoder, supporting for our motivation in adopting the BOW strategy for the image selection.

heading: 3.6 Case Study
text: Figure 5 illustrates two representative NER cases comparing InstructNER, MultiNER, and MAF. Regarding the first example, MultiNER incorrectly labels the named entity Iman Shumpert as O. However, with the aid of the image containing Iman Shumpert, MAF and InstructNER accurately align the text with the corresponding image region, and assign the correct labels. Regarding the second example, both MAF and MultiNER misclassify Loch Lomond, whereas InstructNER leverages attention guidance to focus more on the Loch Lomond lake, leading to the correct classification as LOC.

heading: 4 Related Work

heading: 4.1 Textual NER and MNER
text: Traditional NER methods combine various neural network architectures with a CRF layer (Ratinov and Roth, 2009) to perform sequence labeling (Huang et al., 2015; Ma and Hovy, 2016; Yang et al., 2018). BERT (Devlin et al., 2019), a pretrained language model, has also shown impressive
results. To enhance NER, Zhu and Li (2022) propose boundary smoothing as a regularization technique for span-based NER. To address the issue of lacking context (Zhang et al., 2020; Ju et al., 2020) in text-only NER methods, MNER is introduced by utilizing image information as supplement. (Zhao et al., 2022; Wang et al., 2023a). However, obtaining high-quality images is costly. In this study, we propose utilizing stable diffusion models for image synthesis to provide additional information.

heading: 4.2 Text-Image Alignment
text: Existing text-image alignment approaches can be categorized into explicit and implicit alignment methods. Explicit methods involve extracting object regions from images and aligning them with corresponding words. Zhang et al. (2021b) utilize a visual grounding toolkit (Yang et al., 2019b) to ground sentences to image regions. Jia et al. (2022) further design queries of label types to enhance the association between regions and tokens. However, explicit alignment methods may suffer from error propagation with inaccurate object regions (Yang et al., 2019b). Implicit methods address this issue by using attention mechanisms to adaptively align texts and images. Yu et al. (2020) introduces a multimodal interaction module that integrates transformer layers with cross-modal attention to perform a hierarchical alignment. To eliminate noise, Zong and Sun (2023) aggregates visual features into bottleneck tokens and propagates the refined tokens into alignment. Despite the efficiency of these approaches, attention may lack concentration and contain irrelevant noise. Therefore, we utilize the by-product cross-attention matrix of stable diffusion to guide soft alignment and facilitate attention concentration.

heading: 5 Conclusion
text: In this paper, we propose InstructNER, a comprehensive method that utilizes the image synthesis capability of the stable diffusion model, in order to reframe the textual NER as the MNER task and learn instructive alignment between synthetic images and raw texts. The experimental results demonstrate the effectiveness of synthetic images and the innovative soft attention guidance alignment (AGA) mechanism in improving NER performance. Moreover, we have made available a large-scale, high-quality dataset of synthetic images, which complements existing raw datasets and provides valuable insights for future research endeavors.
Limitations
Our method employs the image synthesis capability of stable diffusion to provide additional context for text-only NER tasks, achieving significant results in general domains. However, domain-specific image synthesis methods, like the medical and electrical domains, encounter challenges in transforming text into images (Kazerouni et al., 2023), resulting in suboptimal outcomes. Regrettably, there is limited current research addressing domain-specific MNER tasks. Therefore, our approach may not be suitable for specialized domains. Furthermore, recent studies have proposed pre-training stable diffusion in specific domains (Moghadam et al., 2023; Kazerouni et al., 2023), which can partially mitigate this issue.

heading: Acknowledgments
text: We thank the anonymous reviewers for their insightful comments. This work was supported by National Science Foundation of China (Grant Nos.62376057) and the Start-up Research Fund of Southeast University (RF1028623234). All opinions are of the authors and do not reflect the view of sponsors.

heading: A Datasets Details
text: We perform experiments on three representative English datasets, Twitter-2015, Twitter-2017 and WikiDiverse, by excluding images that solely consist of textual corpus. Note that WikiDiverse is a cutting-edge multimodal entity linking dataset based on WikiNews. We transform WikiDiverse into a MNER dataset to further verify the effectiveness on the news domain. The statistic are shown in Table 6. Among these, Twitter2015 and Twitter2017 encompass four types of entity categories: Person, Organization, Location, and Other. WikiDiverse contains seven categories, namely: Person, Organization, Country, Movie, Event, Building, and Other. We utilize precision (P), recall (R), and micro F1 score (F1) to evaluate the performance of named entity recognition for all datasets.

heading: Future Work
text: Moreover, we have made available a large-scale, high-quality dataset of synthetic images, which complements existing raw datasets and provides valuable insights for future research endeavors. Limitations
Our method employs the image synthesis capability of stable diffusion to provide additional context for text-only NER tasks, achieving significant results in general domains. However, domain-specific image synthesis methods, like the medical and electrical domains, encounter challenges in transforming text into images (Kazerouni et al. , 2023), resulting in suboptimal outcomes. Regrettably, there is limited current research addressing domain-specific MNER tasks. Therefore, our approach may not be suitable for specialized domains. Furthermore, recent studies have proposed pre-training stable diffusion in specific domains (Moghadam et al. , 2023; Kazerouni et al. , 2023), which can partially mitigate this issue