Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge
Authors: Jinyuan Li
Han Li
Zhuo Pan
Di Sun
Jiahao Wang
Wenkun Zhang
Gang Pan
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal Named Entity Recognition
Information Extraction
Large Language Model
TL;DR: Activating the potential of LLM in MNER task by endowing ChatGPT with reasonable heuristics.

Abstract: Multimodal Named Entity Recognition (MNER) on social media aims to enhance textual entity prediction by incorporating image-based clues. Existing studies mainly focus on maximizing the utilization of pertinent image information or incorporating external knowledge from explicit knowledge bases. However, these methods either neglect the necessity of providing the model with external knowledge, or encounter issues of high redundancy in the retrieved knowledge. In this paper, we present PGIM --- a two-stage framework that aims to leverage ChatGPT as an implicit knowledge base and enable it to heuristically generate auxiliary knowledge for more efficient entity prediction. Specifically, PGIM contains a Multimodal Similar Example Awareness module that selects suitable examples from a small number of predefined artificial samples. These examples are then integrated into a formatted prompt template tailored to the MNER and guide ChatGPT to generate auxiliary refined knowledge. Finally, the acquired knowledge is integrated with the original text and fed into a downstream model for further processing. Extensive experiments show that PGIM outperforms state-of-the-art methods on two classic MNER datasets and exhibits a stronger robustness and generalization capability.

Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge
Jinyuan Li', Han Li,, Zhuo Pan Di Sun', Jiahao Wang', Wenkun Zhang", Gang Pan
"School of New Media and Communication, Tianjin University "College of Mathematics, Taiyuan University of Technology
"College of Intelligence and Computing, Tianjin University
*Tianjin University of Science and Technology
University of Copenhagen
{jinyuanli, wjhwtt, pang edu. cn lihan0928@link tyut. edu. com panzhuotju@@mail. com, dsun@tust. edu. cn, j1s704@alumni. ku. dk
Abstract
impict knowletee
xample Awarencss
Test Coption
Multimodal Named Entity Recognition (MNER) on social media aims enhance textual entity prediction by incorporating image-based clues. Existing studies mainly focus on maximizing the utilization of pertinent image information incorporating external knowledge from explicit knowledge bases. However, these methods either neglect the necessity of providing the model with external knowledge, or encounter issues of high redundancy in the retrieved knowledge. this paper, we present PGIM - two-stage framework that aims to leverage ChatGPT as implicit knowledge base and enable it to heuristically generate auxiliary knowledge for more efficient entity prediction. Specifically, PGIM contains a Multimodal Similar Example Awareness module that selects suitable examples from small number of predefined artificial samples. These examples are then integrated into formatted prompt template tailored to the MNER and guide ChatGPT to generate auxiliary refined knowledge. Finally, the acquired knowledge is integrated with the original text and fed into downstream model for further processing. Extensive experiments show that PGIM outperforms ofte-of-theeart methods on two classic MNER datasets and exhibits stronger robustness and generalization capability.'
CP3s adbrvcit werdoms dur
P
PiDaacannanteer plays wutatin Cle
Text 13 year CP3 Duncan
Baselme: MBCx PER
PGIMIDuns):
Tex: year CP3) CP3) Tim DuncaniPE RR
PER. +PER
Transtormer Embeddings
Figure 1: The "CP3" in the text is class of entities that are difficult to predict successfully by existing stud- ies. PGIM demonstrates successful prediction of such entities with approach more similar to human cogni- tive processes by endowing ChatGPT with reasonable heuristics.
these posts possesses inherent characteristics sociated with social media, including brevity and an informal style writing. These unique char- acteristics pose challenges for traditional named entity recognition (NER) approaches (Chiu and Nichols, 2016; Devlin et al., 2018). To leverage the multimodal features and improve the NER perfor- mance, numerous previous works have attempted to align images and text implicitly using various attention mechanisms (Yu et al., 2020; Sun et al., 2021), but these Image-Text (I+T) paradigm meth- ods have several significant limitations. Limita- tion 1. The feature distribution of different modali- ties exhibits variations, which hinders the model to learn aligned representations across diverse modal- ities. Limitation 2. The image feature extractors used in these methods are trained on datasets like ImageNet (Deng et al., 2009) and COCO (Lin al., 2014), where the labels primarily consist of nouns rather than named entities. There are obvious de- viations between the labels of these datasets and the named entities we aim to recognize. Given
1 Introduction
Multimodal named entity recognition (MNER) has recently garnered significant attention (Lu et al., 2018). Users generate copious amounts of unstruc- tured content primarily consisting of images and text on social media. The textual component in
correspondingg authors. !Our code is publicly available at h/titt/ com/ JinYuanLi@0y 2/PGI/

---

these limitations, these multimodal fusion methods may not be as effective as state-of-the-arl language models that solely focus on text.
in MNER task by endowing ChaiGPT with reason- able heuristics?
In this paper, we present PGIM- - a conceptu- ally simple framework that aims to boost the perfor- mance of model by Prompting ChatGPT In MNER to generate auxiliary refined knowledge. As shown in Figure 1, the additional auxiliary refined knowl- edge generated in this way overeomes the limita- tions of (i) and (ii). We begin by manually annotat- ing limited set of samples. Subsequently, PGIM utilizes the Multimodal Similar Example Aware- ness module to select relevant instances, and seam- lessly integrates them into a meticulously crafted prompt template tailored for MNER task, thereby introducing pertinent knowledge. This approach effectively harnesses the in-context few-shot learn- ing capability of ChatGPT. Finally, the auxiliary refined knowledge generated by heuristic approach of ChatGPT is subsequently combined with the original text and fed into a downstream text model for further processing.
While MNER is a multimodal task, the contri- butions of image and text modalities to this task are not equivalent. When the image cannot pro- vide more interpretation information for the text, the image information can even be discarded and ignored. In addition, recent studies (Wang et al., b021b; Zhang et al., 2022) has shown that intro- ducing additional document-level context on the basis of the original text can significantly improve the performance of NER models. Therefore, re- cent studies (Wang et al., 2021a, 2022a) aim to solve the MNER task using the Text-Text (T+T) paradigm. In these approaches, images reason- ably converted into textual representations through techniques such as image caption and optical char- acter recognition (OCR). Apparently, the inter-text attention mechanism is more likely to outperform the cross-modal attention mechanism. However, existing second paradigm methods still exhibit cer- tain potential deficiencies:
PGIM outperforms all state-of-the-ard models based on the Image-Text and Text-Text paradigms on two classical MNER datasets and exhibits a stronger robustness and generalization capability, Moreover, compared with some previous methods, PGIM friendly to most researchers, its implemen- tation requires only a single GPU and a reasonable number of ChatGPT invocations.
(i) For the methods that solely rely on in-sample information, they often fall short in scenarios that demand additional external knowledge to enhance text comprehension.
(ii) For those existing methods that consider in- troducing external knowledge, the relevant knowledge retrieved from external explicit knowledge base (e.g., Wikipedia) is too redun- dant. These low-relevance extended knowl- edge may even mislead the model's under- standing of the text in some cases.
Related Work
2.1 Multimodal Named Entity Recognition
Considering the inherent characteristics of social media text, previous approaches (Moon et al., 2018; Zheng et al., 2020; Zhang et al., 2021; Zhou et al.. 2022; Zhao et al., 2022) have endeavored to incor- porate visual information into NER. They employ diverse cross-modal attention mechanisms facil- itate the interaction between text and images. Re- cently, Wang et al. (2021a) points out that the per- formance limitations of such methods are largely attributed to the disparities in distribution between different modalities. Despite Wang et al. (2022c) try to mitigate the aforementioned issues by using further refining modal attention, training this und-to-end cross-modal Transformer architectures imposes significant demands on computational sources. Due to the aforementioned limitations. ITA (Wang et al., 2021a) and MoRe (Wang et al., 2022a) attempt to use a new paradigm to address MNER. ITA circumvents the challenge of multi-
Recently, the field of large language models (LLMs) is rapidly advancing with intriguing new findings and developments (Brown et al., 2020; Touvron et al., 2023). On the one hand, recent research on LLMs (Qin al., 2023; Wei et al., 2023; Wang et al.. 2023a) shows that the effect of the generative model in the sequence labeling task has obvious shorteomings. On the other hand, LLMs achieves promising results in various NLP (Vilar et al., 2022; Moslem et al., 2023) and multi- modal tasks (Yang et al., 2022; Shao et al., 2023). These LLMS with in-context learning capability comprehensive representation
can be perceived of internet-based knowledge and can offer high- quality auxiliary knowledge typically. So we ask: Is it possible to activate the potential of CharGPT

---

modal alignment by forsaking the utilization raw visual features and opting for OCR and image cap- tioning techniques to convey image information. MoRe assists prediction by retrieving additional knowledge related to text and images from explicit knowledge base. However, none of these meth- ods can adequately fulfill the requisite knowledge needed by the model to comprehend the text. The advancement of LLMss address the limitations iden- tified in the aforeemnntioned methods. While the direct prediction of named entities by LLMs in the full-shot case may not achieve comparable per- formance to task-specific models, we can utilize LLMs as an implicit knowledge base to heuristi- cally generate further interpretations of text. This method is more aligned with the cognitive and rea- soning processes of human.
based on auxiliary knowledge, PGIM combines the original text with the knowledge information generated by ChatGPT. This concatenated input is then fed into a transformer- encoder to gener- ate token representations. Finally, PGIM feeds the representations into the linear-chain Conditional Random Field (CRF) (Lafferty et al., 2001) layer to predict the probability distribution of the original text sequence (detailed in $3.3). An overview of the PGIM is depicted in Figure 2.
3.1 Preliminaries
Before presenting the PGIM, we first formulate the MNER task, and briefly introduce the in-context learning paradigm originally developed by GPT-3 (Brown et al., 2020) and its adaptation to MNER
Task Formulation Consider treating the MNER task as a sequence labeling task. Given a sentence T (41, ... (nn with n tokens and its correspond- ing image I, the goal of MNER is to locate and clas- sify named entities mentioned in the sentence label sequence y {y ..... In where y/ E Y are predefined semantic categories with the BIO2 tagging schema (Sang and Veenstra, 1999).
2.2 In-context learning
With the development of LLMs, empirical stud- ies have shown that these models (Brown et al., 2020) exhibit an interesting emerging behavior called In-Context Learning (ICL). Different from the paradigm of pre-training and then fine-tuning language models like BERT (Devlin et al., 2018), LLMs represented by GPT have introduced novel in-context few-shot learning paradigm. This paradigm requires no parameter updates and can achieve excellent results with just a few examples from downstream tasks. Since the effect of ICL is strongly related to the choice of demonstration examples, recent studies have explored several ef- fective example selection methods, e.g., similarity- based retrieval method (Liu et al., 2021; Rubin al., 2021), validation set scores based selection (Lee al., 2021), gradient-based method (Wang et al., 2023b). These results indicate that reason- able example selection can improve the perfor- mance of LLMs.
In-context learning in MNER GPT-3 and its successor ChatGPT (hereinafter referred to collec- tively as GPT) are autoregressive language models pretrained on tremendous dataset. During infer- ence, in-context few-shot learning accomplishes new downstream tasks in the manner of text se- quence generation tasks on frozen GPT models. Concretely, given a test input æ, its target y is pre- dicted based on the formatted prompt p(h, C, as the condition, where h refers to a prompt head de- scribing the task and in-context C = {c1, .n)) contains in-context examples. All the h, C, x, y
are text sequences, and target y {v ... y²) is text sequence with the length of L. At each decoding step I, we have:
3 Methodology
y' = argmax argmaxpriM(y P.
gf
PGIM is mainly divided into two stages. In the stage of generating auxiliary refined knowledge, PGIM leverages limited set of predefined artificial samples and employs the Multimodal Similar Ex- ample Awareness (MSEA) module to carefully se- lect relevant instances. These chosen examples then incorporated into properly formatted prompts, thereby enhancing the heuristie guidance provided to ChatGPT for acquiring refined knowledge. (de- tailed in $3.2). In the stage of entity prediction
where LLM represents the weights of the pre- trained large language model, which are frozen for new tasks. Each in-context example e; = (z,, Wi) consists of an input-target pair of the task, and these examples is constructed manually or sampled from the training set.
Although the GPT-42 can accept the input of mul- timodal information, this function is only in the in-
Phttps://openaa hhttps///peena..om///rrottttttt

---

Pro defined art cal samples
Prompt Head
hupurtc
Remeree withie boitns
Visiany Lang wanquugge Pre Pree-trann training Model
with aay Foreviss bee froa cauch
Teet on oun Gas aur luber His senon Answer lamed Seree
Frozen ChatGPT
Vanilla MNER Model
That rntuene
Input sample
thendker mes sesersee wndes
Text The Fremie Leagues batle #champiooss Leagus, Europa qualitiin..
Hec laages tate spprrrrnnnn wnaje Amowen
Input
Auxilary Knowladge
Transformer-baal basedd mbeddings
Ty)
Figure 2: The architecture of PGIM.
ternal testing stage and has not yet been opened for public use. In addition, compared with ChatGPT, GPT-4 has higher costs and slower API request speeds. In order to enhance the reproducibility of PGIM, still choose ChatGPT the main re- search object of our method. And this paradigm provided by PGIM can also be used GPT-4. In order enable ChatGPT to complete the image- text multimodal task, we use advanced multimodal pre-training model to convert images into image captions. Inspired by PICa (Yang et al., 2022) and Prophet (Shao et al., 2023) in knowledge-basee VQA, PGIM formulates the testing input as the following template:
3.2 Stage-1. Auxiliary Refined Knowledge Heuristic Generation
Predefined artificial samples The key to making ChatGPT performs better MNER is to choose suitable in-context examples. Acquiring accurately annotated in-context examples that precisely reflect the annotation style of the dataset and provide a means to expand auxiliary knowledge poses sig- nificant challenge. And directly acquiring such examples from the original dataset is not feasible.
To address this issue, we employ a random sam- pling approach to select a small subset of sam- ples from the training set for manual annotation. Specifically, for Twitter-2017 dataset, we randomly sample 200 samples from training set for manual labeling, and for Twitter-2015 dataset, the number is 120. The annotation process comprises two main components. The first part involves identifying the named entities within the sentences, and the second part involves providing comprehensive justification by considering the image and text content, as well as relevant knowledge. For many possibilities en- counter in the labeling process, what the annotator needs to do is to correctly judge and interpret the sample from the perspective of humans. For sam- ples where image and text related, we directly state which entities in the are emphasized by the image. For samples where the image and text are unrelated, we directly declare that the image description is unrelated to the text. Through artifi-
Text: Z Image: p In Question: In Answer:
where f, p and represent specific test inputs. \n stands for a carriage return in the template. Sim- ilarly, each in-context example e; is defined with similar templates as follows:
Text: in Image: pA Question: q In Answer: a,
where t, Pi. q and a, refer to text-image- question-answer quadruple retrieved from prede- fined artificial samples. The complete prompt tem- plate of MNER consisting of fixed prompt head, some in-context examples, and a test input is fed to ChatGPT for auxiliary knowledge generation.

---

cial annotation process, we emphasize the entities and their corresponding categories within the sen- tences. Furthermore, we incorporate relevant auxil- iary knowledge to support these judgments. This meticulous annotation process serves as a guide for ChatGPT, enabling it to generate highly relevant and valuable responses.
I is the index of top-N similar samples in G. The in-context examples are defined as follows:
( Pj. pJ. bj EI
In order to efficiently realize the awareness of similar examples, all the multimodal fusion fea- tures can be calculated and stored in advance.
Multimodel Similar Example Awareness Mod- ule Since the few-shot learning ability of GPT largely depends on the selection of in-context ex- amples (Liu et al., 2021; Yang et al., 2022), we design Multimodel Similar Example Awareness (MSEA) module to select appropriate in-context examples. As classic multimodal task, the pre- diction of MNER relies on the integration of both textual and visual information. Accordingly, PGIM leverages the fused features of text and image as the fundamental criterion for assessing similar ex- amples. And this multimodal fusion feature can be obtained from various previous vanilla MNER models.
Heuristics-enhanced Prompt Generation After obtaining the in-context example C, PGIM builds a complete heuristics-enhanced prompt to exploit the few-shot learning ability of ChatGPT in MNER. A prompt head, of in-context examples, and testing input together form complete prompt. The prompt head describes the MNER task natural language according to the requirements. Given that the input image and text may not al- ways have a direct correlation, PGIM encourages
ChatGPT to exercise its own discretion. The in- context examples (c..... , of the MSEA module. For test- ing input, the answer slot is left blank for ChatGPT to generate. The complete format of the prompt template is shown in Appendix A.4.
constructed from the results
Denote the MNER dataset D and predefined ar- tificial samples G as:
((4. M
3.3 Stage-2. Entity Prediction based on Auxiliary Refined Knowledge
g [(t..Pj. 15
where ti Pt. Y4 refer to the text, image, and gold labels. The vanilla MNER model M trained on DD mainly consists of backbone encoder M₁ and CRF decoder Mc. The input multimodal image- text pair is encoded by the encoder M3 to obtain multimodal fusion features H::
Define the auxiliary knowledge generated by ChatGPT after in-context learning Z {21, _m), where m is the length of Z. PGIM concatenates the original text (4,,-.. .m)
with the obtained auxiliary refining knowledge as [T; Z] and feeds it to the transformer-based en- coder:
H M)(
{hi,'-. "....,h, An.................. hn+m) embedd 2])
In previous studies, the fusion feature H after cross-attention projection into the high- dimensional latent space was directly input to the decoder layer for the prediction of the result. Un- like them, PGIM chooses H. the judgment basis for similar examples. Because examples approxi- mated in high-dimensional latent space are more likely have the same mapping method and entity type. PGIM calculates the cosine similarity of the fused feature 7. between the test input and each predefined artificial sample. And top-N similar predefined artificial samples will be selected as in- context examples to enlighten ChatGPT generation auxiliary refined knowledge:
Due the attention mechanism employed by the transformer based encoder, the token representa- tion H = {h1,- ... hn obtained encompasses per- tinent cues from the auxiliary knowledge Z. Sim- ilar to the previous studies, PGIM feeds H to a standard linear-chain CRF layer, which defines the probability of the label sequence given the input sentence T:
n II (1i yis hs) 1=1
P 7)
II WEY
+ (1/ hi)
HTH,
= arg TopN JE(1.2....N) |21
where (644-11 3/. h2) and v/6 1: 1/4 hi) are po- tential functions. Finally, PGIM uses the negative

---

log-likelihood (NLL) as the loss function for the input sequence with gold labels y²:
2015), BERT-CRF (Devlin et al., 2018) well as the span-based NER models (e.g., BERT-span, RoBERTa-span (Yamada et al., 2020)), which only consider original text. The second group of meth- ods includes several latest multimodal approaches for MNER task: UMT (Yu et al., 2020), UMGF (Zhang et al., 2021), MNER-QG (Jia et al., 2022), R-GCN (Zhao et al., 2022), ITA (Wang al., 2021a), PromptMNER (Wang al., 2022b), CAT- MNER (Wang et al., 2022c) and MoRe (Wang al., 2022a), which consider both text and correspond- ing images.
(0) log Po(y"(T.Z)
4 Experiments
4.1
Settings
Datasets We conduct experiments on two public MNER datasets: Twitter-2015 (Zhang et al., 2018) and Twitter-2017 (Lu et al., 2018). These two clas- sic MNER datasets contain 1000/1000/3257 and 3373/723/723 (train/developmentttes) image-text pairs posted by users on Twitter.
The experimental results demonstrate the superi- ority of PGIM over previous methods. PGIM sur- passes the previous state-of-the-art method MoRe (Wang et al., 2022a) in terms of performance, This suggests that compared with the auxiliary knowl- edge retrieved by MoRe (Wang et al., 2022a) from Wikipedia, our refined auxiliary knowledge offers more substantial support. Furthermore, PGIM ex- hibits more significant improvement in Twitter- 2017 compared with Twitter-2015. This can be attributed to the more complete and standardized labeling approach adopted in Twitter-0017, con- trast to Twitter-2015. Apparently, the quality of dataset annotation has a certain influence on the ac- curacy of MNER model. In cases where the dataset annotation deviates from the ground truth, accurate and refined auxiliary knowledge leads the model to prioritize predicting the truly correct entities, since the process of ChatGPT heuristically generating auxiliary knowledge is not affected by mislabeling. This phenomenon coincidentally highlights the ro- bustness of PGIM. The ultimate objective of the MNER is to support downstream tasks effectively. Obviously, downstream tasks of MNER expect to receive MNER model outputs that are unaffected by irregular labeling in the training dataset. We further demonstrate this argument through case study, detailed in the Appendix A.3.
Model Configuration PGIM chooses the back- bone of UMT (Yu et al., 2020) as the vanilla MNER model to extract multimodal fusion features. This backbone completes multimodal fusion without too much modification. BLIP-2 (Li et al., 2023) advanced multimodal pre-trained model, is used for conversion from image to image caption. The version of ChatGPT used in experiments is gpt-3.5-turbo and sampling temperature is set to O. For a fair comparison, PGIM chooses to use the same text encoder KLM-RoBERTaanne (Conneau et al., 2019) as ITA (Wang et al., 202 Ia), PromptM- NER (Wang et al., 2022b), CAT-MNER (Wang et al., 2022c) and MoRe (Wang et al., 2022a).
Implementation Details PGIM is trained by Py- torch on single NVIDIA RTX 3090 GPU. During training, we use AdamW (Loshchilov and Hutter, 2017) optimizer to minimize the loss function. We use grid search to find the learning rate for the embeddings within [1 10 5,5 10 $). Due to the different labeling styles of two datasets, the learning rates of Twitter-201 2015 and Twitter-2011 are finally set to 5 x 10-6 and 7 x 10-6, And we also use warmup linear scheduler to control the learning rate. The maximum length of the sentence input is set to 256, and the mini-batch size is set to 4. The model is trained for 25 epochs, and the model with the highest Fl-score on the development set is selected evaluate the performance on the test set. The number of in-context examples N in PGIM is set to 5. All of the results are averaged from 3 runs with different random seeds.
4.3 Detailed Analysis
Impact of different text encoders on perfor- mance As shown in Table 2, We perform ex- periments by replacing the encoders of all XLM- RoBERTatage (Conncau et al., 2019) MNER meth- ods with BERT (Kenton and Toutanova, 2019). BaselinGBERT represents inputting original sam- ples into BERT-CRF. All of the results are av- craged from 3 runs with different random seeds. The marker refers to significant test p-value <
4.2 Main Results
We compare PGIM with previous state-of-the-art approuches on MNER in Table 1. The first group of methods includes BiLSTM-CRF (Huang et al.,

---

Twitter-2015
Twitter-2017
Methods
Single Type(F1)
Overall
Single Type(F1)
Overall
PER LOC ORG OTH.
Prc.
Rec.
F1
PER
L.OC
ORG
OTH.
Pre.
Rec.
F1
Text
BiLSTM-CRFI
76.77
72.56
41.33
26.80
68.14
61.09
64.42
85.12
72.68
72.50
52.56
79.42
73.43
76.31
BERT-CRFI
85.37
81.82
63.26
44.13
75.56
73.88
74.71
90.66
84.89
83.71
66.86
86.10
83.85
84.96
BERT-SPAN
85.35
81.88
62.06
43.23
75.52
73.83
74.76
90.84
85.55
81.99
69.77
85.68
84.60
85.14
RoBERTa-SPAN
87,20
83.58
66.33
50.66
77.48
77.43
77.45
94.27
86.23
87.22
74.94
88.71
89.44
89.06
Text+Image
UMT
85.24
81.58
63.03
39.45
71.67
75.23
73.41
84.73
82.24
70.10
85.28
85.34
85.31
UMGF
84.26
83.17
62.45
42.42
74.49
75.21
74.85
91.92
85.22
83.13
69.83
86.54
84.50
85.51
MNER-QG
85.68
81.42
63.62
41.53
77.76
72.31
74.94
93.17
86.02
84.64
71.8
88.57
85.96
87.25
R-GCN
86.36
82.08
60.78
41.56
73.95
76.18
75.00
92.86
86.10
84.05
72.38
86.72
87.53
87.11
ITA
78.03
89.75
PrompIMNER
78.03
79.17
78.60
89.93
90.60
90.27
CAT-MNER
88.04
84.70
68.04
$2.33
78.75
78.69
78.72
94.61
88.40
88.14
80.50
90.27 90.67
90.47
MoRewa
77.79
89.49
MoReingge
77.57
90.28
MoReMÆ
79.21
90.67
PGIM(Ours)
88.34 84.22 70.15
52.34
79.21
79.45
79.33*
96.46
89.89
89.03
79.62
90.86
92.01
91.43*
10.02
10.12 10.36
+0.98
10663
#0.22
10.06
+0.02
+0.68
+0.53 #2.25
10.16 +0.07 +0.09
Table 1: Performance comparison on the Twitter-15 and Twitter-17 datasets. For the baseline model, results of methods with come from et (2020), and results with come from Wang et al. (2022c). The results of multimodal methods all retrieved from the corresponding original paper. The marker * refers significant test p-value < 0.05 when comparing with CAT-MNER and MoReinage/eer.
Twitter-2015
Twitter-2017
experiment, especially the Twitter-2017 dataset. We think the reasons for this phenomenon are as fol- conceals the defects of previous MNER methods through its strong encod-
Pre.
Rec.
Pre.
Rec.
Fl
lows:
BaselineneRT
75.56
73.88
74.71
86.10
83.85
84.96
UMT!
71.67
75.23
73.41
85.28
85.34
85.31
UMGFF
74.49
75.21
74.85
86.54
84.50
85.51
ing ability, and these defects are further amplified after using BERT For example, the encoding
R-GCNI
73.95
76.18
75.00
86.72
87.53
87.11
ITABERT 1
75.60
85.72
ability of BERT base on long text is weaker than XLM-RoBERTa Tatgee and the additional knowledge retrieved by MoReImage/Test is much longer than PGIM. Therefore, shown in Table 2 and Table
CATBERT
76.19
74.65
75.41
87.04
84.97
85,99
MoRCimge BERI
73.16
74.64
73.89
85.49
86.38
85.94
MoRera: BERT
73.31
74.43
73.86
85.92
86.75
86.34
PGIMAERT
75.84
77.76
76.79*
89.09
90.08
89.58*
5, the performance loss of MoReimageTTes is larger than the performance loss of PGIM after replacing BERT base.
+0.30
+0.22
10.19
+0.24
10.08 +0.10
Table 2: Results of methods with are retrieved from the corresponding original paper. And results with come from Wang et al. (2022c).
Compared with direct prediction of ChatGPT Table 3 presents the performance comparison be- tween ChatGPT and PGIM in the few-shot scenario. VanillaGPT stands for no prompting, and Prompt- GPT denotes the selection of top-N similar samples for in-context learning. As shown in Appendix A.4, employ similar prompt template and utilize the same predefined artificial samples in-context examples for the MSEA module to select from. But the labels these samples are replaced by named entities in the text instead of auxiliary knowledge.
0.05 when comparing with ITA, CAT-MNER and MoReimageTest And represents the results after we replace the text encoder in the MoRe official code with BERT The experimental results show that PGIM achieves greater performance improvement than the XLM-RoBERTaaggs version
http:///gttttcooccc///eeel////// master//example//ooRe

---

Twitter-2015
Twitter-2017
Twitter-2015
Twitter-2017 Rec.
Pre.
Rec.
Pre.
Rec.
Pre.
Rec.
Pre.
F1
fs-10
0.16 12.00 0.32
0.26
12.95
0.51
Baseline
76.45 78.22 77.32
88.46
90.23 89.34
fs-50
50.09 54.50 52.20
49.40
51.96
50.65
w/o MSEAN-1
78.15
79.01
78.58
90.49
90.82
90.65
fs-100
57.33 66.26 61.47
69.51
74.09
71.73
w/o MSEA N=
78.11
79.82
78.95
90.62
91.49
91.05
fs-200
65.16 73.57 69.11
80.97
85.05 82.96
w/o MSEAN 10
78.47
79.21
78.84
90.54
91.77 91.15
full-shot
79.21 79.45 79.33
90.86
92.01 91.43
PGIMN=1
78.40
79.21
78.76
89.90
91,63 90.76
VanillaGPT
42,96 75.37 54.73
52.19
75.03 61.56
PGIMx=6
79.21
79.45
79.33
90.86
92.01 91.43
PromptGPTx=. N
51.96
75.24 61.47
56.99
74.77 64.68
PGIMN=10
78.58
79.67
79.12
90.54
92.08 91.30
PrompiGPT
57.41
73.98 64.65
72.03
75.50 73.73
PromptGPTN=10
58.57 74.07 65.41
72.90
77.65 75.20
Table 4: Effect of the number in-context examples on auxiliary refined knowledge.
Table 3: Comparison of ChatGPT and PGIM in few-
shot case. VanillaGPT and PromptGPT stand for direct prediction using ChatGPT.
its input is the original text that does not contain any auxiliary knowledge. w/o MSEA represents a random choice of in-context examples. All results are averages of training results after three random initializations. Obviously, the addition of auxil- iary refined knowledge can improve the effect of the model. And the addition of MSEA module can further improve the quality of the auxiliary knowledge generated by ChatGPT, which reflects the effectiveness of the MSEA module. An appro- priate number of in-context examples can further improve the quality of auxiliary refined knowledge. But the number of examples is not the more the bet- ter. When ChatGPT is provided with an excessive number of examples, the quality of the auxiliary knowledge may deteriorate. One possible explana- tion for this phenomenon that too many artificial examples introduce noise into the generation pro- cess of ChatGPT. As a pre-trained large language model, ChatGPT lacks genuine comprehension of the underlying logical implications in the examples. Consequently, excessive number of examples may disrupt its original reasoning process.
The BIO annotation method is not considered in this experiment because it little difficult for ChatGPT. Only the complete match will be con- sidered, and only if the entity boundary and entity type are both accurately predicted, we judge it as a correct prediction.
The results show that the performance of Chat- GPT on MNER is far from satisfactory compared with PGIM in the full-shot case, which once again confirms the previous conclusion of ChatGPT on NER (Qin et al., 2023). In other words, when we have enough training samples, only relying on Chat- GPT itself will not be able to achieve the desired ef- fect. The capability of ChatGPT shines in scenarios where sample data scarce. Due the in-context learning ability of ChatGPT, it can achieve signif- icant performance improvement after learning a small number of carefully selected samples, and its performance increases linearly with the increase of the number of in-context samples. We conduct experiments to evaluate the performance of PGIM in few-shot case. For each few-shot experiment, we randomly select 3 sets of training data and train 3 times each set obtain the average result. The results show that after 10 prompts, ChatGPT performs better than PGIM in the fs- 100 scenario on both datasets. This suggests that ChatGPT ex- hibits superior performance when confronted with limited training samples.
Case Study Through some case studies in Fig- ure 3, we show how auxiliary refined knowledge can help improve the predictive performance of the model. The Baseline model represents that no auxiliary knowledge is introduced. MoReText and MoReimage denote the relevant knowledge of the input text and image retrieved using text re- triever and image retriever, respectively. In PGIM, the auxiliary refined knowledge generated by Chat- GPT is structured into components: the first component provides a preliminary estimation of the named entity, and the second component offers a corresponding contextual explanation. In these examples, "Leadership Course". "Big B", "Maxim"
Effectiveness of MSEA Module Table 4 demon- strates the effectiveness of the MSEA module. We use the auxiliary refined knowledge generated by ChatGPT after N in-context prompts to construct the datasets and train the model. The text encoder the Baseline model is XLM-RoBERTaa Tajagee and

---

SNDDE__
Leaderani
Young 1thi 2010 thones.
Chaudhary Holland
Captions: leadenship
holding chilc street.
ALdd bery refored knowedgee
Aucillary refined knowledge
NDCS (orpeiieiao) Bocetion)
Rg peeeerbaberryy
Lu Coeane pewetimannng)
Zalkann Ohn uchery (ueosr))
Young Deod hhople aucherson
Hollnnd ceicountiyy
Doal Cf Birrrtt Soddcy UK, event Locdors/ip
elc mentore Zakwan Chouuhheery dity, willlo Bucly Biand es3odare
(mich yuinnng for young ood pocole image of p000 covorising Joaderthi
Course
6f B. The sonaoncc indicasa that aro Hollund, which
diicity
boctooessuriiy image sooos tolding child buayy
confirmm desall
M005
218000
Label:
Gold
MBC D
Label:
GOLD Raweline
PER FER
Boseliine
Mellinues MICC
O PERJ X
Molleyus: PCCMoum
CI PER (LCC
Mattinn
0
PCclMuur MICC
Nolltyea
O PFR
PF PER 10C%
MA M
Evode? lataa first
BJP cadies Police from march
make
Maxim
Mantralyy Amested. SSack
MAN
Captions:
magaone.
Caption:: people around
Audd Bary rofined knowledee:
Aucillory refined knowledge:
cnilics Elto tactoo
ILP (pelifood parjjf Police
mwer?"
Reasoring: The sontenoo directy ment ond Booo a' krown act0r who 105 appcaros in var ous fimo and locesiio shows. The
Martmbyy (poeettee builing))
Besenting: The ser bence med ona person Sootradeer
iecheologyy bny... and enternaerant The imagevor of iitos the on the cower Khomm sapport prost
cieu Janusa Party
pollccll
tnlla The polce
aru matiooed stopplia the gi procsler fion mar Lowands
rrapertte
mode
bakaya ponommet Lualing
imagl
Labat GOLD
Label: GOLD
CRG LOC
Baneline
PER MISC
Mollawe
PER MISC
Hooeline
LOC ORC LOC
Kallienok PGiffoass
LOGI ORB | CRC .00 V
Mullttee
PJR MISC
PGIK PER PER|ORS ORS V
LOG ORS LOC
Figure Four case studies how auxiliary refined knowledge can help model predictions.
and "Mumbai BJP" are all entities that were not accurately predicted by past methods. Because auxiliary refined knowledge provides explicit explanations for such entities, PGIM makes the correct prediction.
their ability to fully capture all the details of an im- age. This issue may potentially be further resolved in conjunction with the advancement of multimodal capabilities in language and vision models (e.g.. GPT-4).
5 Conclusion
Ethics Statement
this paper, we propose a two-stage framework called PGIM and bring the potential of LLMs to MNER a novel way. Extensive experiments show that PGIM outperforms sfatc-of-the-art meth- ods and considerably overcomes obvious problems in previous studies. Additionally, PGIM exhibits strong robustness and generalization capability, and only necessitates single GPU and reasonable number of ChatGPT invocations. In our opinion, this is an ingenious way of introducing LLMs into MNER We hope that PGIM will serve as a solid baseline to inspire future research on MNER and ultimately solve this task better.
In this paper, we use publicly available Twitter- 2015, Twitter-2017 datasets for experiments. For the auxiliary refined knowledge, PGIM generates them using ChatGPT. Therefore, trust that all data we use does not violate the privacy of any user.
Acknowledgments
This work was supported by the Natural Science Foundation of Tianjin (N.22ICCBBC000400) and by the 2023 CCF-Baidu Songguo Foundation (Re- search Scene Text Recognition Based Pad- dlePaddle).
Limitations
References
In this paper, PGIM enables the integration of mul- timodal tasks into large language model by con- verting images into image captions. While PGIM achieves impressive results, consider this Text- Text paradigm as transitional phase the devel- opment of MNER, rather than the ultimate solution. Because image captions are inherently limited in
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional Istm-cnns. Transac-

---

tions of the association for computational linguistics, 4:357-370.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint 1711v:111..050000
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019, Unsupervised cross-lingual representation learning at scale, arXiv preprint arxiv: 191.0166.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Anmual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers), pages 1990-1999.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEÉ conference on computer vision and pattern recognition, pages 248-255. lece.
Seungwhan Moon, Leonardo Neves, and Vitor Car- valho. 2018. Multimodal named entity recogni- tion for short social media posts. arXiv preprint arXiv:1802.077
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1810.408..
Yasmin Moslem, Rejwanul Haque, and Andy Way. 2023. Adaptive machine translation with large language models. arXiv preprint arX1
Chengwei Qin, Aston Zhang, Zhuosheng Zhang. Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint (XXXi:2332.06477
Zhiheng Huang. Wei Xu, and Kai Yu. 2015. Bidirec- tional Isum-erf models for sequence tagging. arXiv preprint 1rXv:1508.00191
Meihuizi Jia, Lei Shen, Xin Shen, Lejian Liao, Meng Chen, Xiaodong He, Zhendong Chen, and Jiaqi Li. 2022. Mner-qg: An end-to-end mre framework for multimodal named entity recognition with query grounding. arXiv preprint arXiv:2211.1473 1e::22211.1477
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.00%.
Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. arXiv preprint cs/9907006.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of naacI.-III., volume I, page 2.
Zhenwei Shao, Zhou Yu, Meng Wang. and Jun Yu. 2023. Prompting large language models with answer heuris- tics for knowledge based visual question answering. arXiv preprint
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data.
Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fang- sheng Weng. 2021. Rpbert: text-image relation propagation- based bert model for multimodal ner. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13860-13868.
Dong- Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sckiya, Jay Pujara, and Xiang Ren. 2021. Good examples make faster learner: Simple demonstration-bart based learning for low-resource ner. arXiv preprint 1
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint arXiv:2301. 12597.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompt- ing palm for translation: Assessing strategies and performance. arXiv preprint 1:XXi:22211.01100
Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th Furopean Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023a. Gpt-ner: Named entity recognition via large language models. arXiv preprint a334v:2304. 10428.
Xinyi Wang. Wanrong Zhu, and William Yang Wang. 2023b. Large language models are implicitly topic models: Explaining and finding good demon- strations for in-context learning. arXiv preprint arXiv:2301.1191
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiy preprint 2011:20000060000

---

Xinyu Wang, Jiong Cai, Yong Jiang, Pengjun Xie. Kewei Tu, and Wei Lu. 2022a. Named entity and relation extraction with multi-modal retrieval. arXiv preprint 22t::2212.016.11
Xin Zhang, Yong Jiang, Xiaobin Wang, Xuming Hu, Yueheng Sun, Pengjun Xie, and Meishan Zhang. 2022. Dpmain-specific ner via retrieving correlated samples. arXiv preprint Xrv: 12995.
Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang. Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021a. Ita: Image-text alignments for multi-modal named entity recognition. arXiv preprint
Fei Zhao, Chunhui Li, Zhen Wu, Shangyu Xing, and Xinyu Dai. 2022. Learning from different text-image pairs: A relation-enhanced graph convolutional net- work for multimodal ner. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3983-3992.
Xinyu Wang. Yong Jiang, Nguyen Bach, Tao Wang. Zhonggiang Huang. Fei Huang, and Kewei Tu. 2021b. Improving named entity recognition by external context retrieving and cooperative learning. arXiv preprint arXiv:2105.036:
Changmeng Zheng, Zhiwei Wu, Tao Wang. Yi Cai, and Qing Li. 2020. Object-aware multimodal named entity recognition in social media posts with adver- sarial learning. IEEE Transactions on Multimedia, 23:2520-2532.
Xuwu Wang. Junfeng Tian, Min Gui. Zhixu Li, Jiabo Ye, Ming Yan, and Yanghua Xiao, 2022b. Promptmner: Prompt-based entity-related visual clue extraction and integration for multimodal named entity recog- nition. In Database Systems for Advanced Applicu- tions: 27th International Conference, DASFAA 2022, Virtual Event, April 11-14, 2022, Proceedings, Part III. pages 297-305 Springer.
Baohang Zhou, Ying Zhang, Kehui Song, Wenya Guo. Guoqing Zhao, Hongbin Wang, and Xiaojie Yuan. 2022. A span-based multimodal variational autoen- coder for semi- supersieee multimodal named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 6293-6302.
Xuwu Wang, Jiabo Ye, Zhixu Li, Junfeng Tian, Yong Jiang, Ming Yan, Ji Zhang, and Yanghua Xiao. 2022c. Cat-mner: Multimodal named entity recognition with knowledge-refined cross-modal attention. In 2022 IEEE International Conference Multimedia and
A Appendix
A.1
Generalization Analysis
Due to the distinctive underlying logic of PGIM in incorporating auxiliary knowledge to enhance entity recognition, PGIM exhibits a stronger gen- eralization capability that is not heavily reliant on specific datasets. 1iitter-0015-2011 denotes the model is trained on Twitter-2015 and tested on Twitter- 2017, vice versa. The results in Table 6 show that the generalization ability of PGIM is sig- nificantly improved compared with previous meth- ods. This further validates the efficacy and superior- ity of our auxiliary refined knowledge in enhancing model performance.
Expo (ICME), pages 1-6. IEEE
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang. Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302. 10205.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: Deep con- textualized entity representations with entity-aware self-attention. arXiv preprint arXiv:2010.0100
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge- based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 308 1 3089.
A.2 Comparison with MoRe
As the previous of method, MoRe re- trieves relevant knowledge from Wikipedia as- sist entity prediction. We experimentally compare the quality of auxiliary knowledge of MoRe and PGIM. The results are shown in Table 5. The baseline method solely relies on the original text without any incorporation of auxiliary informa- tion. MoReTest and MoRe denote the relevant knowledge of the input text and image retrieved us- ing text retriever and image retriever, respectively. Ave.length represents the average length of the aux- iliary knowledge in entire dataset. Memory indi- cates the GPU memory size required for training model. Ave. Improve represents the average result
Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal trans- former. Association for Computational Linguistics.
Dong Zhang. Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multi- modal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14347-14355.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named en- tity recognition in tweets. Proceedings of the AAAI conference on artificial intelligence, volume 32.

---

Single Type(FI)
Overall
Methods
Ave. length
Memory(MB)
Ave. Improve
PER
LOC
ORG
Pre.
Rec.
F1
Twitter-2015
BaseLine
87.04
83,49
67.34
50.16
76.45
78.22
77.32
11865
86.92
83.08
68.20
49.15
77.12
77.77
77.45
227.41
16759
J. 0.00
MoRéiaage
87.38
83.78
67.75
49.38
77.44
78.06
77.75
203.00
16711
1 0.228
PGIM(Ours)
88.34
84.22
70.15
52.34
79.21
79.45
79.33
104.56
13901
1 1.86
Twitter-2017
BiselLine
95.07
87.22
85.82
78.66
88.46
90.23
89.34
11801
MoRetau
95.16
88.77
87.00
77.71
89.33
90.45
89.89
241.47
16695
0,50
MoReaaage
94.43
87.43
86.22
74.77
88.06
89.49
88.77
192.00
16447
4. 0.80
PGIM(OurS)
96.46
89.89
89.03
79.62
90.86
92.01
91.43
94.52
13279
1 2.07
Table Comparison of MoRe with PGIM. Since the original paper of MoRe (Wang et al., 2022a) did not report its Single Type (FI) the Twitter-2015 and Twitter-2017 datasets, its released code and count the results. All of the results are averaged from 3 runs with different random seeds.
Twitter-001--2200
Twitter-2017-2200
trieved by ChatGPT clearly states that "Mumbai BJP refers to the Bharatiya Janata Party". How- ever, the information retrieved by MoReText from
Pre.
Rec.
F1
Pre.
Rec.
F1
UMT!
67,80
55.23
60.87
64.67
63.59
64.13
UMGFt
69.88
56.92
62.74
67.00
62.81
66.21
Wikipedia provides almost no assistance in recog- nition of named entities. MoRe alleviates this prob- lem to some extent by introducing the Mixture of Experts (MoE) module in the post-processing stage. They fixed the parameters of MоReтeи and MoRelmage- and trained the MoE module for 50 epochs on the basis of them. But shown in Ta- ble before, compared with MoReMot- PGIM still shows better results without any post- post-processing.
CAT-MNER
70.69
59.44
64.58
74.86
63.01
68.43
PGIM
72.66
65.51
68.90
76.13
64.87
70.05
Table 6: Comparison of the generalization ability. For the baseline model, results with i come from Zhang et (2021), and results with come from Wang et al. (2022c).
after summing the improvement of each indica- tor compared with the baseline method. All mod- els use KLM-RoBERTa (Conneau et al., 2019) the text backbone with fixed batch size of 4. The experimental results demonstrate that PGIM achieves performance improvement while requir- ing shorter average auxiliary knowledge length and consuming less memory. This observation high- lights the lightweight nature of PGIM and further underscores the superiority of our auxiliary refined knowledge compared with auxiliary knowledge of MoRe sourced from Wikipedia.
Furthermore, we also show an error prediction of PGIM in Figure In this case, "Bush" is not a named entity that is hard to predict correctly. But since the additional knowledge retrieved by Chat- GPT clearly states that "Bush 41' is name of person, the prediction of PGIM is not in line with the gold label. This illustrates that the additional knowledge retrieved from ChatGPT can affect the final prediction of named entities to some extent. But the reason why MoReText can make correct pre- diction is obviously not related to the knowledge it retrieves from the Wikipedia, because "Bush" is not even mentioned in its knowledge. In fact, by using only the original text after masking the noise retrieved from the Wikipedia, the model can more easily predict correctly.
Additionally, we observe that in certain cases, the introduction of auxiliary knowledge by MoRe can even lead to deterioration in model perfor- mance. One possible explanation for this phc- nomenon is that the information retrieved from Wikipedia often contains redundant or irrelevant content. The first case in Figure illustrates this phenomenon well. In this case, PGIM makes the correct prediction because the information re-
In summary, considering the relevance and length of retrieved information, using Chat- GPT is obviously more suitable for this addi- tional knowledge-basce NER method than using Wikipedia. The information retrieved from Chat-

---

GPT is generally unambiguous and directional. which causes to significantly help predictions in most cases, and may also mislead predictions in rare cases. But the information retrieved from Wikipedia may mislead the original predictions in many cases.
A.3 Predictions for mislabeled examples
We observe that the annotation quality of the Twitter-2015 dataset is suboptimal. There have been large number of errors and omissions in this dataset. This the reason why the accuracy of Twitter-2015 has significantly decreased compared with Twitter-2017. However, as shown in Figure since the first stage of ChatGPT heuristically gen- erating auxiliary knowledge is not affected by mis- labeling, PGIM correctly predicts those unlabeled entities. This also demonstrates the robustness of PGIM. As future direction, intend reanno- the dataset facilitate better development of the MNER task.
A.4 Prompt template
We present the template for prompting ChatGPT to generate answers. In Figure 6, PGIM guides Chat- GPT for auxiliary refined knowledge generation. In-context examples and answers in the template are selected from predefined artificial samples by the MSEA module. In l'igure 7, we guide ChatGPT make direct predictions. In-context examples selected from the same predefined artificial samples by the MSEA module. Note that the answers here are no longer human answers, but named entities in text.

---

Gold labet Mumbal (.F(ORE) eithee sirpowe by Po loe tro march ng sowards MartmiegssLoo Areddas #SackdS made
Gold labe: BURiPERR 41 43 Do Niet Plamilo Encorse to Dan Trump PERG
stapped Polics from marching tauarcs MUCC Amesttd #BackEhinde
McRe Mumba OLC BUPCORS]
# #DECLASSIFY
McRe u Bush o Not Plan Encoans Donaks TrumpPERRy
FGMM Mumbal BUFTORE) of codrre snggaeddiy eng MarralyyeLOOC
PEIN: Bush (PER] No Planto
Arrested #SackShinde
Additional information from MoRa : GEN Ram Nalk in BUP Mumbaal North GEN Fam Naie RUIP 10th ok Sebhi 24, 970689 ok Saeta -Wi--unbbi-hhhi saiishh--ortii North GPN
Additional Information from MoRerues [iaber Ak-A Al-Satar))
-exs wein---eenn---ee---e Jaber al Alhmaa Sabar 128 June 1926 10 January 2005)
Sylvia taken refuge
hideout One of Kids, Denry, returns
(-esArabllc long agge- Arabic-leo: (cimal الرر anl? Jim cyskat the sest Sabahnak- Sabaiiclez dyriady the Enir of Kuwsl and see Commandeer
steplather's apartment bring clothes suspacted of Chempaiin, linoes-rrempeg linceesess, (resAmy AcceerArmy hjjcts prioon Donnal--le-), aor instruction
amested
police,
Ryo
(Rarson ]
chref Comma ndeesige the seriiitary Kuwaato litry Kuwatt-pp who served posts from Docamber 1977
doath Jonuary 2006 to egenclos The this monemch
releazes on immate, Farsons Colin Dernell-
Kinsii Indepe ndence-inde independeneeeeee from Brttn Empine Britann des, Jober prav cusly scrvod so Ministry Finance (Kusaalt ministrr finnnoo and erx comooageee from 1980 to 1965 of win he aopool Primee airisllet.....
Cerebrai
Machine. She gives take he rnflreees governmen men't diiddin posing ng as ne United Stake Deper Iment of un ted States Depertment of Defonse-oog
Additional Information from PGIM:
Named entlles:
1. Sootrachar perton Mumba (lcostio)
Additional Information from PEIM:
3. BJP (po rical party)
Policc onforcoment)
Namodd antiiies: Bush 41 (pensonfor mer proskeent
5. Maaleg ggavanmmnt bakking)
2. Bush 45 gier son't "ommer press den't)
bezcting about a politcall protest Vumboi Mumba Soctradher lieey branch
Ressoning: sertence mentioos a person
2. Donad Trump ipersenpressdrrri cancidate).
the analyy andla Parly, major aditica party in Incla. The polion are mentioned as stopping the protestere from merching lowards Mantralsya apvernment menermmnntt building Looo The hashtag #Sacköninde is lizyy reference to a politicll lssan or con trovans, but without further cortoot is undiaar what this rofers to. The mage of a crowd of people standir sround bus does not appeer to be directly reated to the sentence, but may a part of the larger context of the political protest.
Rossoning: The sontence metioons Bush 41 and 4S, who are both former pces don't of
Uniled Stallos The senleence aloo meriis menionaa Donali To Thump, who is curnent presidential candidate. The to sontance, not montinned antiise..
"declassff"" does seem relaned connocted any the namod
Figure 4: Two case studies on how information retrieved from Wikipedia by MoRe and information retrieved by PGIM from ChatGPT affects model predictions.
Text: RT guoess Twos Injurie fallow crish Invorurta town centre
Texts Bill smill oorrt of Brakine and Johannes Wekiemulle FinnnD Cactions Four people stardingg g twat to paano ronon.
Cartions: police officer stands next chrth Faks baien involved in erish.
lary refined knowlddger
Aud bary knowlddger
Namned witliie Prwcs Jouncl (newn nulee)
Nammed aniitee Bro Renbex (pensong? Ooter Erskiie (peeno))
Recconing Invemate (ecaiin))
leberzee Uenderader (penion))
Fenfibborry (ongan
cullec Era. Btady reporlingon the shad, Innerase
Phoncring The aorannae dana
scaatoo, Bucly laai koom steae the cssed cocou wes. The
Ube poafc the mandicrdd concurts, Puttar Enukano and thomec
inaquu pollr los
to tae anchaciin
Jutuernne wisaderude aru
suspoata
puople ganad coudi lown vantan, Thansloos,
paoshe soluted the sanderca dai razadana pefformnn ui due mun ban sunnoel
lai our hanoe sead Et sragee ara-d boccy slec.
Origitad ineotaaion: PeaMogurs
Invonurte
Oniiral PoDMaur
Frskinn
IOC, vi
Feter Erskine PER
Johannes Welcemuler PER
Text: An Bouth Africs coonorry plips into technical reoessle
Teat: Hove this quote. Robinn Sharms credited for photo. My
library grow librrrgg
Captions: A fiee with ovtr
Captiona:
nary pooale
Ausyilary lany refined knowledge:
"ORDNARY PEOPLE HAVE BIG TVS
Aucilaary refinee knowledge
Abina
EXTRAURDINARY PEDPLE HAVE BIG LIBSRRISS
harnec serill anc Baion Steamn (peraad
Beeasting The bemoe mann or Fouder Sharmse linel parnaon
Ressoning The somtence mentonn alrics country loconed
reacor the placts
spacue closs
bor rign own region disntly rntteed menonnn increseraaton of the courryy ontonn onnor afocted by the
imge siytin
maqqeei
prowine Binary The arapa clecciplioon follin burtaal
Masaho and lie auconill of tawinn a large oree. it rnci cleer
eccnome downttum Dieend the nomed entty bentenoe
the canda, is romod onty the sonto noe la Robinn Shorm..
wit Houee Starres roe the was situ
specito.
credil
associated
Thereforne
Oriiiaaleuntatioo::: PiG Mours
Eo Africa
Original aptotation: PnllMuue"
Robin Bharma
O PFR
Sourth Aficcs 1OC
Briti Stoe
Figure 5: Some mislabeled examples of Twitter-2015 datasets.

---

P Prompt template for ChatG PT to make auxiliary explanation
Here are some content that people post on Twitter, and these content are composed of original text and image descriptions the original text. Please note that the text and image descriptions here may may relevant, make your own judgment, Please follow the data annotation style and method reflected in the example provided, comprehensively analyze the image description and the original text, determine which named entities and their corresponding types are included in the original text, and explain the reason for your judgment. Notice: just in 'Text', not include 'Image descriptions' don't change the writing style and format of entity names, and Words after the @ sign not counted.
Text: #Sports Connor Cook could not satisfy NFL teams' questions about leadership. Image descriptions: Michigan state football player in uniform.
Question: Comprehensively analyze the Text and the image description, which named entities and their corresponding lypes are included in the Text? explain the reason for your judgment.
Answer: Named entities:1. Connor Cook (personvplayer)) 2. NFL (league/organization) Reasoning: The sentence mentions Connor Cook, a former Michigan State football player who was drafted by the NFL. The NFL the highest-level professional football league in the world. The image of a
Text: #NBA Enhanced: OKC Thunder is NOW
Image descriptions: Kevin Durant and Russell Westbrook.
Question: Comprehensively analyze the Text and the Image description. which named entities and their corresponding types are included in the Text? explain the reason for your judgment.
Answer: Named entities:1. OKC Thunder (team/franchise)) 2. NBA (leaguelorganizztion) Reasoning: The sentence mentions OKC Thunder, professional basketball team based Oklahoma City. The NBA mentioned as the organization that the OKC Thunder are playing in, is possible that the text and image are related in the sense that the odds for OKC Thunder to win may have been influenced by their past performances with the team.
Text: The Premier Leagues battle for #ChampionsLeague Europa qualifying. Image descriptions: A soccer player in action the field. Question: Comprehensively analyze Text and Image description, which named entities and their corresponding types are included in Text? explain reason for your judgment.
Answer:
Figure prompt template for ChatGPT to make auxiliary explanation.

---

P Propt template for ChatG PT to direct predict
Here some content that people post Twitter, and these content composed of original text and image descriptions the original Please note that the may not relevant, so make your own judgment. Please follow the data annotation style and method reflected in the example provided, comprehensively analyze the image description and the original text, determine which named entities and their corresponding types are included in the original text. There will only be 4 types of entities: ["LOC, 'MISC, 'ORG', 'PER". Make the answer format like: ('entity name1', "entity type ['entiiy name2', 'entity type2"]
image descriptions here may or
Notice just in 'Text' not include 'Image descriptions', don't change the writting style and format of entity names in original Text, and the words beginning with @ sign are not counted.
Text: #Sports Connor Cook could not satisfy NFL teams' questions about leadership.
Image descriptions: Michigan state football player in uniform. Question: Comprehensively analyze the Text and the Image description, which named entities and their corresponding types are included in the Text? Answer: ['Connor Cook', 'PERJ, [NNFL', 'ORG']
Text: #NBA Enhanced: OKC Thunder is NOW
Image descriptions: Kevin Durant and Russell Westbrook. Question: Comprehensively analyze the Text and the Image description, which named entities and their corresponding types are included in the Text? Answer: ['OKC Thunder'. 'ORG], ['NBA', 'ORG]]
Text: The Premier Leagues battle for (ChampionsLeaguue Europa qualifying. Image descriptions: A soccer player action field.
Question: Comprehensively analyze the Text and the image description, which named entities and their corresponding types are included in the Text?
Answer:
Figure 7: A prompt template ChatGPT to direct predict.


Reviews:
[{"review_id": "kIE2YoefMn", "reviewer": "Reviewer_KB7s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments on our work. And thank you for your affirmation of the pros of our paper. We will address your concerns as follows:\n\n**Q1: As the author mentioned in Model Configuration, they use the XLM-RoBERTa-large as encoder, but as far as I know, most of the previous methods used Bert-base as the encoder, so whether it is a fair comparison is also a concern. I suggest adding experiments with Bert-base as the encoder.**\n\nA1: First, we would like to emphasize the fairness of experimental comparison, the detailed reasons are as follows:\n\nThe experimental results of all the competitive MNER methods (e.g., ITA[1]，PromptMNER[2]，CAT-MNER[3]，MoRe[4]) shown in Table 1 in the past two years, their text encoders are all RoBERTa instead of BERT. In fact, ITA is a dividing line. Starting from the ITA[1] in 2022, MNER research has all chosen RoBERTa as the encoder. It is precisely because of this that, as shown in Table 1, after ITA[1], the accuracy of the MNER methods has achieved a leapfrog performance improvement compared with the \"previous methods used Bert\" you mentioned (e.g., UMT[5], UMGF[6], R-GCN[7]).\n\nAs shown in the \"Text\" part of Table 1, replacing BERT with RoBERTa can achieve a performance improvement of about 3%. Therefore, in order to make a fair comparison, PGIM can only follow the settings of these advanced MNER methods (ITA[1]，PromptMNER[2]，CAT-MNER[3]，MoRe[4]) in the past and use RoBERTa. It is also under this fair experiment with a fixed encoder that PGIM shows better performance than other methods, which reflects the innovation and effectiveness of our method.\n\nHowever, thank you very much for your thoughtful suggestion! We had even more surprising findings in supplementary experiments. We performed experiments by replacing the encoders of all open-source RoBERTa-large MNER methods with BERT-base. Text encoders for all methods in Table 6 are BERT-base. Baseline$_{BERT}$ represents inputting original samples into BERT+CRF. All of the results are averaged from 3 runs with different random seeds. The marker * refers to significant test p-value < 0.05 when comparing with CAT-MNER and MoRe(Image/Text). The experimental results show that PGIM not only continues SOTA performance, but also achieves a greater performance improvement than the RoBERTa version experiment, especially on the Twitter-2017 dataset.\n\n|           Table 6     |  |**Twitter-2015**     |     | |    **Twitter-2017**  |     |\n|-----------------------|:----------------:|:---:|:---:|:----------------:|:---:|:---:|\n|                       |       Pre.       | Rec.|  F1  |       Pre.       | Rec.|  F1  |\n| Baseline$_{BERT}$ |      75.56      |73.88|74.71 |      86.10       |83.85|84.96 |\n| UMT                   |      71.67      |75.23|73.41 |      85.28       |85.34|85.31 |\n| UMGF                  |      74.49      |75.21|74.85 |      86.54       |84.50|85.51 |\n| R-GCN                 |      73.95      |76.18|75.00 |      86.72       |87.53|87.11 |\n| ITA$_{BERT}$   |        -        |  -  |75.60 |        -         |  -  |85.72 |\n| CAT-MNER$_{BERT}$ |    **76.19**    |74.65|75.41 |      87.04       |84.97|85.99 |\n| MoRe$_{Image BERT}$ |  73.16  |74.64|73.89 |      85.49       |86.38|85.94 |\n| MoRe$_{Text BERT}$ |  73.31  |74.43|73.86 |      85.92       |86.75|86.34 |\n| PGIM$_{BERT}$  |      75.84      |**77.76**|**76.79*** |     **89.09**      |**90.08**|**89.58*** |\n|                       | $\\pm$0.30 |$\\pm$0.22|$\\pm$0.19 | $\\pm$0.24 |$\\pm$0.08|$\\pm$0.10|\n\nWe think the reason for this phenomenon is as follows: RoBERTa conceals the defects of previous MNER methods through its strong encoding ability, and these defects are further amplified after using BERT. For example, the encoding ability of BERT on long text is weaker than RoBERTa, and the additional knowledge retrieved by MoRe (Image/Text) is much longer than PGIM. Therefore, as shown in Table 4 and Table 6, the performance loss of MoRe (Image/Text) is larger than the performance loss of PGIM after replacing BERT.\n\nWe will add this experiment and analysis to the experimental section of paper. In addition, in order to avoid similar doubts for readers, we will add a supplementary statement in the 4.1 Model Configuration chapter -- \"For a fair comparison, PGIM chooses to use the same text encoder XLM-RoBERTa-large as ITA[1], PromptMNER[2], CAT-MNER[3] and MoRe[4].\"\n\n**Q2: As shown in Table 1, the improvement of some results over the compared baseline MoRe is not so great, so I would like to see the standard deviation of these results. Besides, no significance tests are conducted.**\n\nA2: First, your insights on standard deviation and significance testing are quite valuable. According to statistics, the standard deviation of PGIM in Table 1 are as follows. And the marker * refers to significant test p-value < 0.05 when comparing with CAT-MNER and MoRe(MoE):  \n\n| Tabel 1${_{add SD}}$ | ******|Twitter|-   2015 |******|******|******|****** | ******|Twitter|- 2017  |****** |****** |****** | ******|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | | Single|  Type(F1)| | |Overall | | |Single |Type(F1) | | Overall | | |\n| | PER | LOC  | ORG | OTH. | Pre. | Rec. | F1 | PER | LOC | ORG | OTH. | Pre. | Rec. | F1 |\n| | | | | | | |Text+Image | | | | | | | |\n| UMT | 85.24    | 81.58 | 63.03 | 39.45 | 71.67 | 75.23 | 73.41 | 91.56 | 84.73 | 82.24 | 70.10 | 85.28 | 85.34 | 85.31 |\n| UMGF | 84.26 | 83.17 | 62.45 | 42.42 | 74.49 | 75.21 | 74.85 | 91.92 | 85.22 | 83.13 | 69.83 | 86.54 | 84.50 | 85.51 |\n| MNER-QG | 85.68 | 81.42 | 63.62 | 41.53 | 77.76 | 72.31 | 74.94 | 93.17 | 86.02 | 84.64 | 71.83 | 88.57 | 85.96 | 87.25 |\n| R-GCN | 86.36 | 82.08 | 60.78 | 41.56 | 73.95 | 76.18 | 75.00 | 92.86 | 86.10 | 84.05 | 72.38 | 86.72 | 87.53 | 87.11 |\n| ITA | - | - | - | - | - | - | 78.03 | - | - | - | - | - | - | 89.75 |\n| PromptMNER | - | - | - | - | 78.03 | 79.17 | 78.60 | - | - | - | - | 89.93 | 90.60 | 90.27 |\n| CAT-MNER | 88.04 | **84.70** | 68.04 | 52.33 | 78.75 | 78.69 | 78.72 | 94.61 | 88.40 | 88.14 | **80.50** | 90.27 | 90.67 | 90.47 |\n| MoRe${_{MoE}}$ | - | - | - | - | - | - | 79.21 | - | - | - | - | - | - | 90.67 |\n| PGIM(Ours) | **88.34** | 84.22 | **70.15** | **52.34** | **79.21** | **79.45** | **79.33*** | **96.46** | **89.89** | **89.03** | 79.62 | **90.86** | **92.01** | **91.43*** |\n| | $\\pm$0.02 | $\\pm$0.12 | $\\pm$0.36 | $\\pm$0.98 | $\\pm$0.63 | $\\pm$0.22 | $\\pm$0.06 | $\\pm$0.02 | $\\pm$0.68 | $\\pm$0.53 | $\\pm$2.25 | $\\pm$0.16 | $\\pm$0.07 | $\\pm$0.09 |\n\nIn fact, we have indeed attempted to display the standard deviation in the Table 1, as it holds significant importance. But considering that most of the methods in Table 1 (UMT[5], UMGF[6], MNER-QG[8], ITA[1], PromptMNER[2], CAT-MNER[3], MoRe[4]) did not report the standard deviation in their paper, we ended up following this \"convention\" in MNER. \n\nThis problem exists in almost all the work in MNER field. But we hope that PGIM can be the terminator of this problem, as a solid baseline in the field of MNER. So, we will modify Table 1 in the camera-ready version to clearly present the standard deviation and significance test results.\n\nSecond, we need to point out that the fair opponent of PGIM should be MoRe (Image/Text) in the Appendix Table 4, because they have the same macrostructure, same parameter quantity and same encoder (RoBERTa-large). The only difference is the method of obtaining additional information and the quality of the acquired knowledge. \n\nWhat Table 1 really wants to reflect is: as we stated in lines 833-846 of the paper, even when making an unfair comparison with MoRe (MoE), which is built upon MoRe (Image+Text) by incorporating post-processing steps-MoE (This means that MoRe[4] needs to train three models: MoRe(Image)+MoRe(Text)+MoRe(MoE)). PGIM still achieves better results without any excessive complex design (Just need to train one model).\n\nIn other words, compared with MoRe (Image/Text) with the same architecture and parameter quantity, PGIM has significant performance advantages (as shown in Table 4). Compared with the complete MoRe (MoE), PGIM still achieves better results with a simpler and lighter model.\n\nYour suggestion also made us realize that when we compare PGIM with MoRe(Image/Text) or MoRe(MoE), we should explain the relationship between them more clearly. Above analyzes will eventually be condensed and added to the Appendix of the paper in camera-ready version.\n\n**Q3: As shown in Table 3, the authors explore the effect of the number of in-context examples on the test set rather than on the dev set, which brings bias to the proposed model.**\n\nA3: First, throughout this paper, the metrics reported by PGIM are the results of the test set. For all methods (UMT[5], UMGF[6], R-GCN[7], MNER-QG[8], ITA[1], PromptMNER[2], CAT-MNER[3], MoRe[4]) compared with PGIM, their reports are also test set results. And most researchers agree that reporting test set accuracy is considered convincing in most deep learning tasks. Given the writing conventions in the field, we did not report the metrics on the dev set in paper.\n\nHowever, out of respect to the reviewer, we recounted the effect of the number of in-context examples on the dev set. Since only the hyperparameters yielding the highest accuracy on the dev set are employed for a single test on the test set and given that the disparity between the dev and test data in both datasets used in our experiments is not substantial, the secondary results do not exhibit a significant distinction. The observed phenomena in the results still adhere to the viewpoints we conveyed in the original text, spanning lines 521 to 548.\n\n\n|         Tabel 7          | Twitter-2015 |       |       | Twitter-2017 |       |       |\n|-------------------|--------------|-------|-------|--------------|-------|-------|\n|                   | Pre.         | Rec.  | F1    | Pre.         | Rec.  | F1    |\n| Baseline          | 76.25        | 78.22 | 77.22 | 88.75        | 90.56 | 89.64 |\n| w/o MSEA$_{N=1}$  | 77.90        | 78.83 | 78.36 | 90.44        | 90.79 | 90.61 |\n| w/o MSEA$_{N=5}$  | 78.34        | 79.32 | 78.83 | 90.54        | 91.77 | 91.15 |\n| w/o MSEA$_{N=10}$ | 77.47        | 79.90 | 78.66 | 90.59        | 91.93 | 91.26 |\n| PGIM$_{N=1}$      | 77.44        | 80.18 | 78.79 | 90.34        | 91.41 | 90.88 |\n| **PGIM$_{N=5}$**  | 78.41        | **80.53** | **79.45** | 90.67        | **92.82** | **91.73** |\n| PGIM$_{N=10}$     | **78.69**    | 79.77 | 79.23 | **90.97**    | 92.07 | 91.52 |\n\n\nThe above constitutes our complete response to the concerns you raised, with the hope of resolving any uncertainties you had. We would greatly appreciate it if you could further engage in a more comprehensive objective assessment of the contributions and the soundness of this paper. Because as far as we know, PGIM represents the pioneering attempt in the era of LLMs to incorporate LLM for addressing MNER challenges. We contend that this work possesses considerable inspiration for the pertinent field and it deserves to be discovered by more NER researchers.\n\n**Reference:**\n\n[1] ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition (NAACL 2022)\n\n[2] PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition (DASFAA 2022)\n\n[3] CAT-MNER: Multimodal Named Entity Recognition with Knowledge-Refined Cross-Modal Attention (ICME 2022)\n\n[4] Named Entity and Relation Extraction with Multi-Modal Retrieval (EMNLP 2022)\n\n[5] Improving multimodal named entity recognition via entity span detection with unified multimodal transformer (ACL 2020)\n\n[6] Multi-modal graph fusion for named entity recognition with targeted visual guidance (AAAI 2021)\n\n[7] Learning from Different text-image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER (ACMMM 2022)\n\n[8] MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query Grounding (AAAI 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper is well-organized and well-written.\n\n(2) They design a new framework to deal with the MNER task, i.e., leverage ChatGPT to generate auxiliary knowledge for more efficient entity prediction.\n\n(3) The experimental analysis and comparison are comprehensive, and the results are competitive.\n", "reasons_to_reject": "(1) As the author mentioned in Model Configuration, they use the XLM-RoBERTa_large as encoder, but as far as I know, most of the previous methods used Bert-base as the encoder, so whether it is a fair comparison is also a concern. I suggest adding experiments with Bert-base as the encoder.\n\n(2) As shown in Table 1, the improvement of some results over the compared baseline MoRe is not so great, so I would like to see the standard deviation of these results. Besides, no significance tests are conducted.\n\n(3) As shown in Table 3, the authors explore the effect of the number of in-context examples on the test set rather than on the dev set, which brings bias to the proposed model.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the author argued that existing studies mainly focus on maximizing the utilization of pertinent image information or incorporating external knowledge from explicit knowledge bases. However, these methods still suffer from two problems: (1) neglect the necessity of providing the model with external knowledge; (2) high redundancy in the retrieved knowledge. To tackle the above issues, they proposed a two-stage framework PGIM that aims to leverage ChatGPT to heuristically generate auxiliary knowledge for more efficient entity prediction. Experiments on two MNER datasets show their model achieves state-of-the-art performance. Further analysis verifies the effectiveness of the method.\n\npros:\n\n(1) Well-organized and well-written.\n\n(2) New framework.\n\n(3) sufficient experimental analysis.\n\ncons:\n\n(1) Unfair comparison.\n\n(2) marginal improvement compared with MoRe and the standard deviation is not provided.\n\n(3) some results have bias.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QUyo2cpMLe", "reviewer": "Reviewer_eSeH", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments on our work. And thank you for your affirmation of the research motivation and comprehensiveness of the experiment. In the following, we reply to specific points raised by the reviewer.\n\n**Q1: The description of the notation process of predefined artificial samples is rather brief and lacks clarity. Such as, (1) How many predefined artificial samples were used? (2) How did the authors annotate samples in the dataset where image and text exhibit low correlation?**\n\nA1: First, about predefined artificial samples: specifically, for Twitter-2017 dataset, we randomly sample 200 samples from the training set for manual labeling, and for Twitter-2015 dataset, the number is 120. The annotators consist of three NLP graduate students with backgrounds in NER and LLMs. For many possibilities encountered in the labeling process, what the annotator needs to do is to correctly judge and interpret the sample from the perspective of humans. For samples where images and text are related, we directly state which entities in the text are emphasized by the image. For samples where the image and text are unrelated, we directly declare that the image description is unrelated to the text. We select a few examples as follows:\n_______________________________________________________________________________________________\nSample ID: 16_05_24_110\n\nNamed entities: 1. The Cure (band) \nReasoning: The sentence mentions \"The Cure\" which is a famous rock band. **The image may not be related to the band the sentence mentioned.**\n\nSample ID: O_1313\n\nNamed entities: 1. Justin Bieber (person/celebrity) 2. New York City (place/city) \nReasoning: The sentence mentions Justin Bieber, a well-known celebrity and musician. New York City is a well-known city in the United States and is mentioned because that is where Bieber was spotted. **The picture has nothing to do with the sentence.** \n_______________________________________________________________________________________________\nAdditionally, after the anonymity period is over, we will upload all paper-related data (including code, the dataset constructed using GPT, and predefined artificial samples) to GitHub for open access. Currently, you can view it through Supplementary Materials.\n\nYour valuable comments made us realize that a more detailed explanation of the notation process of predefined artificial samples can better help readers understand paper. So, the above replies will eventually be condensed in the camera-ready version and added to paragraph 3.2 Predefined artificial samples. \n\n**Q2: The findings presented in Table 3 suggest that the MSEA module has minimal impact on the model's performance, raising doubts about its utility. The observed performance improvement in the model can likely be attributed to the utilization of predefined artificial samples and a well-designed prompt.**\n\nA2: First, the effectiveness of the MSEA module is positively correlated with the quantity and quality of predefined artificial samples. The presence or absence of the MSEA module will lead to the utilization of distinct predefined samples for constructing prompts for GPT. When randomly selecting samples to feed into GPT, the significance of these samples primarily lies in guiding GPT to generate responses with a controllable format. The difference from the former is that the most similar samples selected by the MSEA module are more likely to assist GPT not only in attending to the response format but also in roughly predicting the correct entities. This is because the answer is highly likely to be embedded within the predefined samples retrieved by MSEA. Evidently, the greater the number of predefined artificial samples, the more likely the MSEA module is to create a scenario where the answer is hidden within the prompt. \n\nIn summary, we reasonably believe that increasing the number of manually labeled samples will make the MSEA module more effective. However, since the original intention of PGIM is to allow the ability of LLM to be used reasonably in MNER and to explore the possibility of obtaining marginal benefits in performance through the MSEA module + a small number of predefined artificial samples. Therefore, after the experiments in Table 3 are sufficient to prove that the introduction of MSEA not only does not damage the accuracy, but also brings a certain degree of accuracy improvement, we did not further invest too many artificial samples. Because we think that the small improvement in accuracy is far less important than the research motivation we want to convey. However, how to make the prompt module such as MSEA produce greater effect with less cost is a new problem worth studying in the future.\n\nFinally, it is worth noting that whether it is the predefined artificial samples, well-designed MNER prompts, or the design of the MSEA module, all are part of the contribution of this paper. The importance of the three parts may be different, but the combination of the three is the final contribution that this paper wants to convey.\n\n**Q3: The author mentions the Baseline in line 526, but there is no detailed explanation of its structure. Although lacking auxiliary knowledge, the Baseline still demonstrates commendable performance.**\n\nA3: This Baseline refers to: compared with the complete PGIM, auxiliary knowledge is not added in any way, and only the original sentence is retained. Therefore, in order to control variables, the other settings of Baseline are exactly the same as PGIM, including using XLM-RoBERTa-large as the text encoder. And as shown in the \"Text\" section in Table 1, replacing BERT with RoBERTa can achieve a 3% performance improvement. This may be an important reason for your mention of \"commendable baseline performance\".\n\nThe reason why the reviewers have this doubt may be that the expression in line 526 of the paper is not clear enough. We will simply modify line 526 of the paper to make the expression clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a novel method for MNER and provides comprehensive experimental validation of PGIM. Leveraging ChatGPT for auxiliary knowledge generation results in improved generalization and robustness.\n2. The experiments were conducted in a fairly comprehensive manner.\n3. The motivation behind this approach, aiming to enhance the performance of MNER using external knowledge, is intuitive and well-founded.\n", "reasons_to_reject": "1. The description of the notation process of predefined artificial samples is rather brief and lacks clarity. Such as,\n(1) How many predefined artificial samples were used?\n(2) How did the authors annotate samples in the dataset where image and text exhibit low correlation?\n\n2. The findings presented in Table 3 suggest that the MSEA module has minimal impact on the model's performance, raising doubts about its utility. The observed performance improvement in the model can likely be attributed to the utilization of predefined artificial samples and a well-designed prompt.\n\n3. The author mentions the Baseline in line 526, but there is no detailed explanation of its structure. Although lacking auxiliary knowledge, the Baseline still demonstrates commendable performance.\n", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper presents a two-stage framework that utilizes ChatGPT to generate auxiliary knowledge, enhancing the accuracy of Multimodal Named Entity Recognition (MNER). The paper conducts sufficient experiments, but some essential elements, such as the annotation process and examples of predefined artificial samples, lack detailed explanations. Additionally, the MSEA module proposed in the paper seems to have minimal impact on model performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "2EiYYYWfye", "reviewer": "Reviewer_hWkn", "summary": "", "questions": "\nPlease see the prior section for my questions/concerns", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for all the highly constructive feedback you've provided, and we particularly appreciate your thorough review of our paper. We will address your concerns as follows:\n\n**Q(A): The paper doesn’t mention how many examples they needed to manual annotate per training dataset. This is one of the most important parts of the paper and needs to be provided so people can understand the tradeoff. The method is well justified and simple, but…**\n\nA: First, about predefined artificial samples: specifically, for Twitter-2017 dataset, we randomly sample 200 samples from the training set for manual labeling, and for Twitter-2015 dataset, the number is 120. The annotators consist of three NLP graduate students with backgrounds in NER and LLMs. For many possibilities encountered in the labeling process, what the annotator needs to do is to correctly judge and interpret the sample from the perspective of humans. For samples where images and text are related, we directly state which entities in the text are emphasized by the image. For samples where images and text are not related, we directly point out that the two are not related.\n\nSecond, although the MNER work in the past two years has rarely open-sourced data and codes, after the anonymity period is over, we will open source all the data of PGIM (codes, datasets, predefined artificial samples) to GitHub. Currently, you can view all the data and code (including predefined artificial samples) related to PGIM through Supplementary Materials.\n\nYour valuable comments made us realize that a more detailed explanation of the notation process of predefined artificial samples can better help readers understand paper. So, the above replies will eventually be condensed in the camera-ready version and added to paragraph 3.2 Predefined artificial samples. \n\n**Q(B): Their results against MoRe are fairly close ( .5 and .3 ) and the appendix comparison ( which is not mentioned in the body of the paper ) doesn’t compare against the full MoRe but rather just how well it does if it only uses one modality ( text or image ). I think there is a probably a good case to make ( efficiency, cost, controllability, etc vs annotation costs ) for why to use PGIM over MoRe, but they should articulate it in the paper and compare more truly along each entity type if possible ( and not just the final F1 scores )**\n\nA: First, we need to point out that the fair opponent of PGIM should be MoRe (Image/Text) in the Appendix Table 4. Because they have the same macrostructure, same parameter quantity and same encoder (RoBERTa-large). The only difference is the method of obtaining additional information and the quality of the acquired knowledge. What Table 1 really wants to reflect is: as we stated in lines 833-846 of the paper, even when making an unfair comparison with MoRe (MoE), which is built upon MoRe (Image+Text) by incorporating post-processing steps-MoE (This means that MoRe[4] needs to train three models: Image+Text+MoE). PGIM still achieves better results without any excessive complex design (Just need to train one model).\n\nTherefore, compared with the complete MoRe (MoE), the primary reason for choosing PGIM is that even though the performance of MoRe (MoE) is competitive, the training difficulty of PGIM and the requirements for hardware are much lower than that of MoRe (MoE). In fact, the parameter quantity of PGIM is the same as that of MoRe(Image) or MoRe(Text). This means that the model parameters of the complete MoRe (MoE), are nearly three times that of PGIM (Image module + Text module + MoE module). Obviously, the training difficulty of MoRe (MoE) and the requirements for hardware far exceed that of PGIM. In addition, MoRe[4] did not open source the data and code of its MoE part. And in the past two years, there has been a lack of open source research in the field of MNER. We hope that PGIM will be easy to use and become a solid baseline for the future of the field of MNER.\n\nSecond, when making a fair comparison (same architecture and model parameters) with publicly available MoRe (Image/Text), the key reasons for choosing PGIM are as follows: as shown in Table 4, compared with MoRe (Image/Text), PGIM achieves a more significant performance improvement (about 2%) with a shorter appended text length on all entity types. In addition, it is worth mentioning that the shorter additional text length in PGIM naturally results in a smaller memory requirement. Less than 16GB training memory requirement (in condition without changing float precision and batchsize) means that PGIM can be trained and fine-tuned on a wider range of GPUs than MoRe (Image/Text) (e.g., V100 16GB/4080 16GB). This lightweight advantage of PGIM is not present in all MoRe (Image/Text/MoE).\n\nBut, for PGIM and MoRE (Image/Text) with similar architectures, what about using a more lightweight text encoder? As requested by the reviewer KB7s, we also conducted an experiment. Text encoders for all methods in Table 6 are BERT-base. Baseline$_{BERT}$ represents inputting original samples into BERT+CRF. All of the results are averaged from 3 runs with different random seeds. The marker * refers to significant test p-value < 0.05 when comparing with CAT-MNER and MoRe(Image/Text). When we switch the text encoder from RoBERTa-large to BERT-base, PGIM gains a much larger lead (about 3%~4%) over MoRe(Image/Text).  This is because BERT's long-text encoding capability is significantly inferior to that of RoBERTa. Consequently, the disadvantage of MoRe's excessively long additional text is further magnified in this context. \n\n|           Table 6     |  |**Twitter-2015**     |     | |    **Twitter-2017**  |     |\n|-----------------------|:----------------:|:---:|:---:|:----------------:|:---:|:---:|\n|                       |       Pre.       | Rec.|  F1  |       Pre.       | Rec.|  F1  |\n| Baseline$_{BERT}$ |      75.56      |73.88|74.71 |      86.10       |83.85|84.96 |\n| UMT                   |      71.67      |75.23|73.41 |      85.28       |85.34|85.31 |\n| UMGF                  |      74.49      |75.21|74.85 |      86.54       |84.50|85.51 |\n| R-GCN                 |      73.95      |76.18|75.00 |      86.72       |87.53|87.11 |\n| ITA$_{BERT}$   |        -        |  -  |75.60 |        -         |  -  |85.72 |\n| CAT-MNER$_{BERT}$ |    **76.19**    |74.65|75.41 |      87.04       |84.97|85.99 |\n| MoRe$_{Image BERT}$ |  73.16  |74.64|73.89 |      85.49       |86.38|85.94 |\n| MoRe$_{Text BERT}$ |  73.31  |74.43|73.86 |      85.92       |86.75|86.34 |\n| PGIM$_{BERT}$  |      75.84      |**77.76**|**76.79*** |      **89.09**      |**90.08**|**89.58*** |\n|                       | $\\pm$0.30 |$\\pm$0.22|$\\pm$0.19 | $\\pm$0.24 |$\\pm$0.08|$\\pm$0.10|\n\nThird, the results of Table 4 and Table 6 also show an interesting result: As described in lines 833-846 of the paper, the performance of MoRe (Image/Text) in some cases is even worse than that of Baseline without any auxiliary knowledge. In other words, using MoRe (Image/Text) to retrieve the additional knowledge from the wiki is not even as good as using the original text alone. The information retrieved by MoRe from wiki is messy, invalid and redundant, which leads to the fact that the original text features finally fed to the decoder contain too much irrelevant noise. This is also one of the core arguments of PGIM, as effective as possible and short text additional knowledge is really effective for NER.\n\nIn addition, according to your valuable suggestions, we also made statistics on the prediction results of PGIM and MoRe (Text) on all entity categories, the results are as follows: \n\n| Tabel 7     | Error$_{MoRe}$ | Right$_{PGIM}$ | Accurate | Error$_{PGIM}$ | Right$_{MoRe}$ | Accurate |\n|-------|-----------------------|-----------------------|----------|-----------------------|-----------------------|----------|\n|  |                       |                       |   **Twitter-2017**       |                       |                       |          |\n| PER   | 69                    | 36                    | 52.17%   | 59                    | 15                    | 25.42%   |\n| LOC   | 30                    | 14                    | 46.66%   | 23                    | 6                     | 26.08%   |\n| ORG   | 116                   | 54                    | 46.55%   | 84                    | 25                    | 29.76%   |\n| OTHER | 87                    | 38                    | 43.68%   | 60                    | 4                     | 0.07%    |\n| |                       |                       |      **Twitter-2015**     |                       |                       |          |\n| PER   | 335                   | 129                   | 38.51%   | 246                   | 40                    | 16.26%   |\n| LOC   | 320                   | 87                    | 27.19%   | 316                   | 71                    | 22.47%   |\n| ORG   | 426                   | 99                    | 23.24%   | 417                   | 84                    | 20.14%   |\n| OTHER | 819                   | 169                   | 20.64%   | 738                   | 88                    | 11.92%   |\n\nSince the performance of MoRe(Text) and MoRe(Image) are similar, considering the space reason, only the comparison results between PGIM and MoRe(Text) are listed here. **Error(X)** represents the number of wrong predictions made by the **X** model, **Right(Y)** represents the number of correct predictions made by the **Y** model among all samples where the **X** model made wrong predictions, and **Accurate=Right(Y)/Error(X)**. The results show that on the 2017 dataset, PGIM is nearly twice as accurate as MoRe when facing all error-prone samples of PER, LOC, and ORG. In the face of the most difficult OTHER category, the accuracy rate has achieved a crushing advantage. In other words, in the face of PGIM's error-prone samples, MoRe basically cannot predict correctly, and in the face of MoRe's error-prone samples, PGIM can make correct predictions in half of the cases. On the 2015 dataset with relatively irregular labels, PGIM still won the overall lead in four categories. Although the lead has diminished somewhat, the accuracy of PGIM still remains twice that of MoRe when facing the PER and OTHER categories.\n\nFinally, we need to point out: the annotation cost of PGIM is extremely low. In Table 3, w/o MSEA$_{N=1,5,10}$ shows the performance of PGIM on randomly selected N artificial samples, and its performance is at the same level as that of the full MoRe (Image+Text+MoE). In other words, when faced with a new dataset, it only needs to simply and quickly construct a few artificial samples adapted to the new dataset to obtain the strong performance shown by PGIM (w/o MSEA). The reason for more artificial samples is to explore the possibility of obtaining marginal gains in performance through the MSEA module + artificial samples. Annotation costs are negligible when there is no need to pursue extreme performance.\n\nYour suggestion made us realize that it is very important to further explain the advantages of PGIM over MoRe(Image/Text/MoE) in the paper. All of the above analyzes will eventually be condensed and added to the Appendix of the paper in camera-ready version.\n\n**Q(C): It would be nice to see how this method would do using an open source alternative (alpaca-lora, vicuna-13B, llama2, etc) and not just ChatGPT for comparison.**\n\nA: Thank you for your valuable comments. We have tried to use multimodal LLMs such as MiniGPT-4[8]/mPLUG-Owl[9], and we finally chose ChatGPT mainly because of the following two points: \n\nFirst, we found that the most obvious defect of this kind of open source alternative compared to ChatGPT is that the operability of its prompt project is much lower than that of ChatGPT/GPT4. The effect of a randomly selected prompt + example is not much worse than that of a well-designed prompt + example. This is contrary to the starting point of PGIM, which aims to explore a prompting approach tailored for MNER. \n\nSecond, when it comes to comparing PGIM with LLMs for direct prediction of named entities. For open source alternatives that can be deployed locally, it is difficult for us to control its persistence to generate answers in a specified format. In the actual test, about 5% of chatgpt's answers did not follow the format in the prompt, and this value was even close to 30% on mPLUG-Owl[9] and MiniGPT-4[8]. Irregular answers will make it impossible to use the program to count the correct rate of the results. Obviously, ChatGPT is the most powerful LLM, and PGIM is far better than ChatGPT, which is enough to achieve the goal of the experiment in Table 2.\n\nYour opinion has inspired us very well. Due to time constraints, we cannot complete the evaluation of various open source LLMs within five days, because it requires a huge workload. We believe that the exhaustive experiments in this paper have provided sufficient evidence for our claim. But we will do our best to test before camera ready. Perhaps in MNER task, how to effectively prompt on LLMs that can be deployed locally such as llama2, is another question that is worthy of our exploration in the future.\n\n**Q(D): Its great that all experiments are averaged over 3 runs, but including error bars would be useful to determine the variance of things (even if only in the appendix for space reasons)**\n\nA: You've presented excellent insights. According to statistics, the standard deviation of PGIM in Table 1 are as follows: \n\n| Tabel 1${_{add SD}}$ | ******|Twitter|-   2015 |******|******|******|****** | ******|Twitter|- 2017  |****** |****** |****** | ******|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | | Single|  Type(F1)| | |Overall | | |Single |Type(F1) | | Overall | | |\n| | PER | LOC  | ORG | OTH. | Pre. | Rec. | F1 | PER | LOC | ORG | OTH. | Pre. | Rec. | F1 |\n| | | | | | | |Text+Image | | | | | | | |\n| UMT | 85.24    | 81.58 | 63.03 | 39.45 | 71.67 | 75.23 | 73.41 | 91.56 | 84.73 | 82.24 | 70.10 | 85.28 | 85.34 | 85.31 |\n| UMGF | 84.26 | 83.17 | 62.45 | 42.42 | 74.49 | 75.21 | 74.85 | 91.92 | 85.22 | 83.13 | 69.83 | 86.54 | 84.50 | 85.51 |\n| MNER-QG | 85.68 | 81.42 | 63.62 | 41.53 | 77.76 | 72.31 | 74.94 | 93.17 | 86.02 | 84.64 | 71.83 | 88.57 | 85.96 | 87.25 |\n| R-GCN | 86.36 | 82.08 | 60.78 | 41.56 | 73.95 | 76.18 | 75.00 | 92.86 | 86.10 | 84.05 | 72.38 | 86.72 | 87.53 | 87.11 |\n| ITA | - | - | - | - | - | - | 78.03 | - | - | - | - | - | - | 89.75 |\n| PromptMNER | - | - | - | - | 78.03 | 79.17 | 78.60 | - | - | - | - | 89.93 | 90.60 | 90.27 |\n| CAT-MNER | 88.04 | **84.70** | 68.04 | 52.33 | 78.75 | 78.69 | 78.72 | 94.61 | 88.40 | 88.14 | **80.50** | 90.27 | 90.67 | 90.47 |\n| MoRe${_{MoE}}$ | - | - | - | - | - | - | 79.21 | - | - | - | - | - | - | 90.67 |\n| PGIM(Ours) | **88.34** | 84.22 | **70.15** | **52.34** | **79.21** | **79.45** | **79.33** | **96.46** | **89.89** | **89.03** | 79.62 | **90.86** | **92.01** | **91.43** |\n| | $\\pm$0.02 | $\\pm$0.12 | $\\pm$0.36 | $\\pm$0.98 | $\\pm$0.63 | $\\pm$0.22 | $\\pm$0.06 | $\\pm$0.02 | $\\pm$0.68 | $\\pm$0.53 | $\\pm$2.25 | $\\pm$0.16 | $\\pm$0.07 | $\\pm$0.09 |\n\nIn fact, we have indeed attempted to display the standard deviation in the Table1, as it holds significant importance. But considering that most of the methods in Table 1 (UMT[5], UMGF[6], MNER-QG[7], ITA[1], PromptMNER[2], CAT-MNER[3], MoRe[4]) did not report the standard deviation in their paper, we ended up following this \"convention\" in MNER.\n\nIn addition, it is very worth mentioning that ITA[1] pointed out that since the baseline method actually selects the test set for parameter adjustment and reports the best result instead of the average result, therefore the final report of ITA[1] is the best accuracy rather than the average accuracy of multiple experiments. Accordingly, this may be the reason why the research in the field of MNER recently (UMT[5], UMGF[6], MNER-QG[7], ITA[1], PromptMNER[2], CAT-MNER[3], MoRe[4]) does not report the standard deviation. \n\nThis problem exists in almost all the work in MNER field. But we hope that PGIM can be the terminator of this problem, as a solid baseline in the field of MNER. So we will modify the results in the camera ready version to clearly present the standard deviation.\n\n**Q(E): For the case study, it would be interesting to do an error analysis against MoRe results in particular (when is it right and PGIM is wrong and vice versa, when is having Wiki access advantageous vs ChatGPT, etc)**\n\nA: We analyzed the advantages of PGIM from the level of statistical indicators in **Q(B)**. In case the statistics are not intuitive enough, we will illustrate through a more intuitive case analysis. \n\nThe first is a situation where MoRe(Text) makes a mistake but PGIM gets it right when faced with the complex sample. For the fourth most complex sample in Figure 4 of paper, here is the additional information MoRe(Text) retrieved from the wiki and the predictions it made:\n_______________________________________________________________________________________________ \n**Additional information from wiki:** GEN Ram Naik M BJP Mumbai-North GEN Ram Naik M BJP 10th Lok Sabha 24.979599 Lok Sabha <hit>Mumbai</hit>-North [ GEN ] Sylvia has taken refuge in the gang's hideout. One of the Kids, Danny, returns to her stepfather's apartment to bring some clothes for her. He is arrested by the police, suspected of the murder. [ 'Neath Brooklyn Bridge ] [ / (Person of Interest) ] in <e:Champaign, Illinois>Champaign, Illinois</e>, Root (<e:Amy Acker>Amy Acker</e>) hijacks a prison bus and releases an inmate, Billy Parsons (<e:Colin Donnell>Colin Donnell</e>), per instruction from the Machine. She then gives him a fake ID so he infiltrates a government building posing as a <e:United States Department of Defense>United States Department of Defense</e> and receives a note with an address. She then leaves Billy to be arrested by the police. Radio <e:Stolytsia>Stolytsia</e> reported that Berkut riot police stopped a motorcade of protesters from heading towards the <e:Mezhyhirya Residence>presidential mansion in Mezhyhirya</e>, a suburb north of Kyiv. [ 1 December 2013 Euromaidan riots ] [ 1 July police stabbing ] [ 12 June 2019 Hong Kong protest ] [ 10 Rillington Place ] [ 1 July police stabbing ] [ 1 July police stabbing ] GEN Anna Joshi M BJP Pune GEN Anna Joshi M BJP 10th Lok Sabha 13.849536 Lok Sabha Pune [ GEN ]\n\n**MoRe prediction:** **Mumbai[LOC] &#x2716;** **BJP[ORG] &#x2716;** cadres stopped by Police from marching towards **Mantralaya[LOC] &#x2714;**. Arrested. #SackShinde\n\n**Gold label:** **Mumbai BJP[ORG]** cadres stopped by Police from marching towards **Mantralaya[LOC]**. Arrested. #SackShinde\n_______________________________________________________________________________________________ \n\nYou read that right, this is not garbled characters, this is indeed the data we sampled from the MoRe(Text) dataset. And the following is the additional information retrieved by PGIM from the GPT:\n\n_______________________________________________________________________________________________ \n**Additional information from GPT:** Named entities:1. Sootradhar (person/handle) 2. Mumbai (location)  3. BJP (political party) 4. Police (law enforcement) 5. Mantralaya (government building) 6. #SackShinde (hashtag/issue)\n\nReasoning: The sentence mentions a person or handle named Sootradhar who is likely tweeting about a political protest in Mumbai. **The Mumbai BJP refers to the local branch of the Bharatiya Janata Party, a major political party in India.** The police are mentioned as stopping the protesters from marching towards **Mantralaya, a government building in Mumbai.** The hashtag #SackShinde is likely a reference to a political issue or controversy, but without further context it is unclear what this refers to. The image of a crowd of people standing around a bus does not appear to be directly related to the sentence, but may be a part of the larger context of the political protest.\n\n**PGIM prediction:** **Mumbai BJP[ORG] &#x2714;** cadres stopped by Police from marching towards **Mantralaya[LOC] &#x2714;**. Arrested. #SackShinde \n\n**Gold label:** **Mumbai BJP[ORG]** cadres stopped by Police from marching towards **Mantralaya[LOC]**. Arrested. #SackShinde\n_______________________________________________________________________________________________ \n\nObviously, the reason why PGIM can make a correct prediction is that the information retrieved by GPT clearly states that \"Mumbai BJP refers to the Bharatiya Janata Party\". However, the information retrieved by MoRe(Text) from Wiki provides almost no assistance in the recognition of named entities.\n\nThen there is a case where MoRe(Text) made the right prediction but PGIM got it wrong:\n\n_______________________________________________________________________________________________\n**Additional information from wiki:** [Jaber Al-Ahmad Al-Sabah] <e:Sheikh>Sheikh</e> Jaber al-Ahmad al-Sabah (29 June 1926 – 15 January 2006) (<e:Arabic language>Arabic</e>: الشيخ جابر الأحمد الجابر الصباح) of the <e:Al-Sabah>al-Sabah</e> dynasty was the Emir of Kuwait and <e:Commander-in-chief>Commander</e> of the <e:Military of Kuwait>military of Kuwait</e> who served in the posts from 31 December 1977 until his death on 15 January 2006 due to <e:Cerebral hemorrhage>cerebral hemorrhage</e>. The third monarch to rule Kuwait since its <e:Independence>independence</e> from <e:British Empire>Britain</e>, Jaber had previously served as <e:Ministry of Finance (Kuwait)>minister of finance and economy</e> from 1962 to 1965 when he was appointed <e:Prime minister>prime minister</e> prior to becoming Kuwait's ruler. [newline_symbol] Main article: <e:Gulf War>Gulf War</e> [Nuclear Security Summit] The Nuclear Security Summit (NSS) is a world summit, aimed at preventing <e:Nuclear terrorism>nuclear terrorism</e> around the globe. The first summit was held in <e:Washington, D.C.>Washington, D.C.</e>, United States, on April 12–13, 2010. The second summit was held in <e:Seoul>Seoul</e>, South Korea, in 2012. The third summit was held in <e:The Hague>The Hague</e>, Netherlands, on March 24–25, 2014. The fourth summit was held in Washington, D.C. on March 31–April 1, 2016 [newline_symbol] Main article: <e:2010 Nuclear Security Summit>2010 Nuclear Security Summit</e>\n\n**MoRe prediction:** **Bush[PER] &#x2714;** 41, 43 Do Not Plan to Endorse **Donald Trump[PER] &#x2714;**.\n\n**Gold label:** **Bush[PER]** 41, 43 Do Not Plan to Endorse **Donald Trump[PER]**.\n_______________________________________________________________________________________________\n**Additional information from PGIM:** Named entities: 1. Bush 41 (person/former president) 2. Bush 43 (person/former president) 3. Donald Trump (person/presidential candidate).\n\nReasoning: **The sentence mentions Bush 41 and 43, who are both former presidents of the United States.** The sentence also mentions Donald Trump, who is a current presidential candidate. The image of the word \"declassify\" does not seem to be related to the sentence, as it is not mentioned in the text or connected to any of the named entities. \n\n**PGIM prediction:** **Bush 41[PER] &#x2716;** , 43 Do Not Plan to Endorse **Donald Trump[PER] &#x2714;**.\n\n**Gold label:** **Bush[PER]** 41, 43 Do Not Plan to Endorse **Donald Trump[PER]**.\n_______________________________________________________________________________________________\nIn this case, Bush is not a named entity that is hard to predict correctly. But since the additional knowledge retrieved by GPT clearly pointed out that Bush 41 is a name of person, PGIM ended up making a prediction that didn't match the gold label. This shows that the knowledge retrieved by PGIM from GPT may mislead the final prediction of the model in a few cases. On the one hand, even with numerous interventions, it is still difficult to make GPT always generate answers that conform to the annotation style of the dataset; on the other hand, in rare cases, GPT may generate wrong answers.\n\nBut the reason why MoRe(Text) can make correct predictions is obviously not related to the information it retrieves from the wiki, because \"Bush\" is not even mentioned in its additional knowledge. In fact, after masking the noise retrieved from the wiki, RoBERTa+CRF can more easily predict correctly. \n\nIn summary, considering the relevance and length of retrieved information, GPT has advantages over wiki in almost any case. The information retrieved from GPT is usually unambiguous and directional, which leads to it providing obvious help for prediction in most cases, but also potentially misleading prediction in rare cases.\n\nFor reasons of space, we do not list further cases. All of the above analyzes will eventually be condensed and added to the paper. And we will select more similar cases and add them to the end of the Appendix of paper to help readers have a more intuitive feeling for the advantages of PGIM.\n\nFinally, we have fixed all the typos grammar style and presentation improvements. Thank you for your careful reading of our article. Your comments are very helpful for the improvement of our paper!\n\nReference:\n\n[1] ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition (NAACL 2022)\n\n[2] PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition (DASFAA 2022)\n\n[3] CAT-MNER: Multimodal Named Entity Recognition with Knowledge-Refined Cross-Modal Attention (ICME 2022)\n\n[4] Named Entity and Relation Extraction with Multi-Modal Retrieval (EMNLP 2022)\n\n[5] Improving multimodal named entity recognition via entity span detection with unified multimodal transformer (ACL 2020)\n\n[6] Multi-modal graph fusion for named entity recognition with targeted visual guidance (AAAI 2021)\n\n[7] MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query Grounding (AAAI 2023)\n\n[8] Minigpt-4: Enhancing vision-language understanding with advanced large language models （arXiv:2304.10592）\n\n[9] mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality （arXiv:2304.14178）", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  The authors present a simple and effective method to get state of the art results on the MNER task\n2.  The ablations are thorough and justify many of the algorithmic decisions made.\n", "reasons_to_reject": "(A) The paper doesn’t mention how many examples they needed to manual annotate per training dataset ( but we know that its greater than 200 .. so possibly 300, 500, 1k, 4k? ).  \n      This is one of the most important parts of the paper and needs to be provided so people can understand the tradeoff ( simplicity vs annotation effort).  \n      The method is well justified and simple, but the way in which they annotate and how many instances they need to annotate is opaque, in particular the second part which “involves providing comprehensive justification by considering the image and text content, as well as relevant knowledge.”.  Are there any examples of this annotation that you can share?  Will this data be provided?\n  \n(B) Their results against MoRe are fairly close ( .5 and .3 ) and the appendix comparison ( which is not mentioned in the body of the paper ) doesn’t compare against the full MoRe but rather just how well it does if it only uses\none modality ( text or image ).  I think there is a probably a good case to make ( efficiency, cost, controllability, etc vs annotation costs ) for why to use PGIM over MoRe, but they should articulate it in the paper and compare more truly along each entity type if possible ( and not just the final F1 scores )\n\n(C) It would be nice to see how this method would do using an open source alternative ( alpaca-lora, vicuna-13B, llama2, etc) and not just ChatGPT for comparison.  \n\n(D) Its great that all experiments are averaged over 3 runs, but including error bars would be useful to determine the variance of things ( even if only in the appendix for space reasons )\n\n(E) For the case study, it would be interesting to do an error analysis against MoRe results in particular ( when is it right and PGIM is wrong and vice versa, when is having Wiki access advantageous vs ChatGPT , etc)\n\n", "questions_to_authors": "\nPlease see the prior section for my questions/concerns", "paper_topic_and_main_contributions": "This paper deals with the task of Multimodal Named Entity Recognition and proposes a two-stage framework PGIM that (1) aims to leverage ChatGPT as an implicit knowledge base and (2) utilize this generated auxiliary knowledge for MNER.  \nThe first step utilizes manually annotated “artificial samples” from within a task’s training dataset and then for each data instance, identifies the K most similar artificial samples and uses those samples as in-context examples (text, image caption, gold labels) that get passed via a prompt template into ChatGPT along with the instance’s text and image, and a pre-defined prompt head (same for all instances). In the second step, the auxiliary refined knowledge output of the first stage, which is ChatGPTs few shot entities prediction for an input instance along with its reasoning, is fed into a transformer-based encoder along with the instance’s text and this output embedding is passed into standard linear-chain CRF layer for the final MNER prediction.  The transformer/CRF layer is fine tuned on the training data using negative log likelihood and then during inference time on test instances is used to make the final prediction.  \n\nThe authors show that over two datasets ( Twitter 2015 and 2017 ) PGIM provides better performance than the current state of the art ( MoRe ) and run experiments/ablations to understand (1) the importance of how many artificial samples are annotated in total for stage 1, (2) how prompting GPT with top N similar samples ( without the subsequent transformer/CRF ) compares to PGIM and(3) the effect of random in-context examples vs the ones selected by PGIM (for different size n in [1,5,10]).  They finally share some qualitative examples to show how auxiliary refined knowledge can help model predictions (compared against a baseline model and MoRe )\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\n- Line 214, the first time you mention CRF you should spell out Conditional Random Field and cite it.\n- The end of the Task formulation sentence “y \\elem Y with Y which predefined .. “ (228-231) is a little weirdly worded.  Maybe “y \\elem Y are predefined … “\n- Line 819 small typo (Ave Imporve -> Ave Improve)"}}]

Author rebuttal:

Meta-review: This paper presents a two-stage framework PGIM that utilizes ChatGPT to generate auxiliary knowledge for Multimodal Named Entity Recognition (MNER). Experimental analysis and comparison are comprehensive. Important details, such as the annotation process and examples, are missing in the submitted version (provided in the rebuttal). The performance gains of the MSEA module are somehow slight.
Decision: Accept-Findings