In-context Learning for Few-shot Multimodal Named Entity Recognition
Authors: Chenran Cai
Qianlong Wang
Bin Liang
Bing Qin
Min Yang
Kam-Fai Wong
Ruifeng Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: In-context Learning
Few-shot Multimodal Named Entity Recognition

Abstract: Thanks in part to the availability of copious annotated resources for some entity categories, existing studies have achieved superior performance in multimodal named entity recognition (MNER). However, in the real-world scenario, it is infeasible to enumerate all entity categories in advance. Therefore, in this paper, we formulate a new few-shot multimodal named entity recognition (FewMNER) task, which aims to effectively locate and identify named entities for a text-image pair only using a small number of labeled examples. Further, we explore the merit of in-context learning (ICL) and propose a novel framework to deal with FewMNER, where three points are taken into account: i.e., converting visual modality, selecting useful examples, and designing an effective task demonstration. Specifically, we first employ an image caption model to convert images into textual descriptions, enabling large language models to absorb information from visual modality. Then, we use the ranking of the sum of similarity rankings from both text and image modalities to select k-nearest examples, which form a demonstration context. Finally, we utilize the MNER definition and the meaning of each entity category as effective instruction. Extensive experimental results demonstrate that our framework outperforms baselines under several few-shot settings.

In-context Learning for Few-shot Multimodal Named Entity Recognition
Chenran Cai Qianlong Wang 12 Bin Liang 1.244 Bing Qin',
Min Yang", Kam-Fai Wong', Ruifeng Xu3.23
Harbin Institute of Technology, Shenzhen, China
? GGangddong Provincial Key Laboratory of Novel Security Intelligence Technologies
*Peng Cheng Laboratory, Shenzhen, China
"The Chinese University of Hong Kong, Hong Kong, China
Research Center for SCIR, Harbin Institute of Technology, Harbin, China SSIAT, Chinese Academy of Science, Shenzhen, China
crcai@stu. hit. edu. cn, qlwang1 1oorutggooooooooooooo com, bin. .ian.gggeu edu. hk, qinbeir hit. edu. cn, min.yangesiat.ar ac cn, kfwongêse. cuhk, edu. hk, xuruifeng@hit. en
Abstract
Hove (Allshsbes
Thanks in part to the availability of copious annotated resources for some entity categories, existing studies have achieved superior perfor- mance in multimodal named entity recognition (MNER). However, in the woal-world scenario, it is infeasible enumerate all entity categories advance. Therefore, in this paper, we formu- late few-shot multimodal named entity recognition (FewMNER) task, which aims to effectively locate and identify named entities for text-image pair only using small num- ber of labeled examples. Further, we explore the merit of in-context learning (ICL) and pro- pose novel framework to deal with F-wM- NER, where three points are taken into account: ie., converting visual modality, selecting use- ful examples, and designing an effective task demonstration. Specifically, we first employ an image caption model to convert images into tex- tual descriptions, enabling large language mod- els absorb information from visual modality. Then, use the ranking of the sum of similar- ity rankings from both text and image modali- ties select k-nearest examples, which form a demonstration context. Finally, we utilize the MNER definition and the meaning each entity category effective instruction. Exten- sive experimental results demonstrate that our framework outperforms baselines under several few-shot settings.
[Lar Ihtesjes onzent Montay
Back shac!
Barelooa Chowned 2016 La Lip Charrptons, Storer
The World Cup the Hggest soccer taemanentt
(PER) ['aane]] LOC! It. YRGY [Burceione La Lige] MISC: DI
1IER Ik "LOC ORG:IL MISCY P World Cuefll
(a) MNER task.
(b) 2-shot MNER task.
Figure I: Illustration of MNER and 2-shot MNER tasks.
"Suarez" (PER), "Barcelona" and "La Liga" (ORG), to finish the MNER task. Most existing methods commonly employ pre-trained models followed by fine-tuning to accomplish the MNER task and achieve superior performance (Lu et al., 2018 8;; Yu et al., 2020; Zhang et al., 2021; Chen et al., 2022b). In terms of existing research efforts, their supe- rior performance generally relies sufficient an- notated data, which is time-consuming and labor- intensive. addition, in practice, entity categories will continue to emerge rather than remain fixed. Therefore, it is impractical to define all entity cate- gories in advance.
To address these issues, motivated by the few- shot Named Entity Recognition (FewNER) task that involves learning unseen entity categories from small number of labeled examples (Fritzler et al., 2019), we extend the MNER task to the few-shot field, named the few-shot multimodal named entity recognition (FewMNER) task, which aims to locate and identify named entities for text-image pair only using a small number of labeled examples. As illustrated in Figure I(b), the 2-shot MNER task
1 Introduction
Multimodal Named Entity Recognition (MNER) aims to identify named entities of different cate- gories from the text with extra image assistance. Consider the example in Figure I(a), we need to recognize three named entities from the text,
"Corresponing author

---

aims accomplish the MNER task based on two labeled text-image pair examples.
Then, we pack selected top-k examples into the demonstration template in ascending order of sim- ilarity rank, such that the most similar example is nearest to the current test sample. Finally, we concatenate instruction, demonstration, and the test sample as the input and feed it into LLM to obtain the prediction output.
Further, to address the FewMNER task, propose leveraging the powerful in in-context learn- ing (ICL) capability of the large language model (LLM). Specifically, we argue that this paradigm can provide a promising direction for solving the FewMNER task by learning from few examples in context without training. However, there are three problems while solving FewMNER using the in-context learning paradigm: (i) For the FewM- NER task, each sample is represented by textual and visual modalities, while the input of LLM is limited to natural language. Thus, we first require seeking ways to convert visual modality into natu- ral language form. (ii) The key to performing ICL is to select few examples to form a demonstra- tion context. Although there are some example selection studies (Chen al., 2022a; Min al., 2022) targeting text classification tasks, selecting some useful examples for ICL in multimodal sce- narios has not been approached. (iil) addition, good demonstration designing precisely is essential to obtain satisfactory performance. Unlike simple classification tasks, the task instruction and output format of MNER need to be constructed according to the extractive nature.
The contributions of this paper are as follows:
We are the first to extend the MNER task to the few-shot field and explore the potential of the in-context learning paradigm for this task.
To adapt the in-context learning paradigm to the FewMNER task, we address three related problems and propose a framework to accom- plish this task.
Through comparison with previous competi- tive methods, our framework exhibits signif- icant advantage in this task. We also conduct extensive analysis experiments reveal the impact of various factors on its performance and provide novel insights for future research.
2
Related Work
2.1 Multimodal Named Entity Recognition
Multimodal Named Entity Recognition (MNER) aims to discover named entities in the unstructured text and classify them into pre-defined types with the help of an auxiliary image. Existing studies could be divided into two categories: cross-modal basedd methods and image conversion- based methods. The former tends to carry out cross- modal interaction using an attention mechanism and to combine textual representation with image representation for MNER For example, some stud- ies (Lu et al., 2018; Moon et al., 2018; Zhang et al.. 2018) first applied LSTM and CNN to extract text and image features, respectively. Then attention is adopted to fuse two modal features to derive textual representation in order to complete entity labeling. In addition to modeling the interaction between text and images, a few studies (Chen et al., 2022b; Zhang et al., 2021) leveraged the semantic correlation between tokens and object regions to derive the final token representations for MNER The latter (Chen et al., 2021; Wang al., 2022b) first aims at converting images and extracting tex- tualized information from them such as captions in order to align image features to the text space. Then this textualized information derived from an
To apply ICL to solve the FewMNER task, we propose corresponding solutions to the above- mentioned problems. First, we employ an image caption model (Wang al., 2022a) to generate textual descriptions from images, which not only converts images into natural language form but also aligns image features into the text space. Second, for selecting examples, we design an efficient sort- ing algorithm based on image and text similarity ranks, which can mitigate the similarity bias caused by different modality models. Then, we utilize this algorithm select top-k examples with the high- est similarity the current test sample. Third, the demonstration design consists of two parts: instruc- tion construction and demonstration construction (Dong et al., 2023). The former aims to inform LLM about the current task. To provide more de- tailed information, we add the description of entity category meaning to the instruction. The latter is define the demonstration template and order se- lected examples into the demonstration template. For the demonstration template, consists of three components: image description, sentence, and output, where output is the label information.

---

Retrieve Example Module
Demonstration Designing Module
You are a sman and imelligmi Multimodal Named Enery Recogittio
system provide you definituon enattie....
New Pied: Blacklum Festivall of Virie 2017
Eatily Datantion:
1 PER: Shoer name or full name of pemon from ary prographic regione."
2. DRG: An orginized group of people with pericoke purpos:
Barcelona Crowned 2016 Lige Champarte Suaree Sink
LOC: Name of geogmaphi location"
4 MISC: Name entriies ther not belong presious thnn groupe"
Fasmples
New French Quarter Festival
Inage description: group of soccer players bugging each other.
Sentonor: Buachin Chowned 2016 La Liga Clanpions Suuzz Sisk
Ouipue; (WER' [Starer] (ORG [Pereloon OC ((ood[ MMSCC
Erie
Image deverjation: soceer player yollow
utumph way
sas
Oupec: (PER (PER ['Maasi] JORG (Leieste |LOC (MNoc MMSSC [NNoue))
Lekeeseer crtumph is why love Soocball say5 Mess.
Image deceijotinn soceer player kicking soceet ball fild.
Sentence: The Wodld Cup the biggeit soccer kossament the world.
Large Language Model
Predict Module
World the higgeet
ball Bold,
world...
Freeze
Irage Deseription
Imago
1PER" I'None",, ORRG (RRGeE 'LOC" "LOL":''Moe" MISC. I'Worid Cupll)
Test Sumple
Figure 2: The architecture of our framework on the 2-shot MNER task.
image is concatenated with the input text to yield the final token representation for completion entity recognition. Despite their promising results, they generally depend on large amount of annotated data, which inadequate in generalizing the abil- ity to locate and identify entities to unseen entity categories.
labeled examples to detect series of entities in t and classify them into pre-defined categories. Following most existing in-context learning work (Dong et al., 2023), we formulate this task as a generation task. A large language model M takes a generation sequence of the maximum score as prediction output conditioning on the context C. For the k-shoot MNER task, C con- tains instruction I and k examples, where C' { s (01, t1, ......,,).. (0k, te, yk)), is demon- stration template and {yI,.... .... Wk is set of free text phrases as the label. Therefore, for the given test sample x = {v, l]. the prediction output ý can be expressed as:
2.2 In- In-Context Learning
With the scaling of the pre-trained model from 110M parameters (Devlin et al., 2019) to over 500B parameters (Smith et al., 2022), the ability of the model has been greatly improved, especially the understanding ability, fluency, and quality of gener- ation. Many studies have demonstrated that large language models (LLMs) have shown an in-context learning ability (Brown et al., 2020), which learn- ing from a few context examples without training. Although various LLMs (e.g-, GPT-3, ChatGPT) have been trained, they are all closed-source and only accessible internally or via paid API services. How to effectively utilize the in-context learning ability of LLMs is an important question. Recently, some studies (Sun et al., 2022; Hu et al., 2022; Zhang et al., 2022) treat LLMs as a service and utilize the in-context learning ability to finish the few-shot and even zero-shot tasks.
max PM C,
(1)
3.2 Overall Architecture
Figure 2 illustrates the overall architecture of our framework for the 2-shot MNER task, which con- tains three main components: (1) Retrieve example module, which utilizes k-nearest neighbors of text and image to select examples. (2) Demonstration designing module, which includes instruction con- struct and demonstration construct. (3) Predict module, which applies large language model to generate prediction results without training.
3.3 Retrieve Example Module
3 Methodology
Previous works (Rubin et al., 2022; Liu et al., 2022) have demonstrated that selecting similar examples to the current test sample can enhance the perfor- mance of LLM. However, these methods only con-
3.1 Task Definition
Given a text é and its correlated image v as input, the fewMNER task only applies small number of

---

sider textual similarity scores, which are insuffi cient for the FewMNER task due to the multimodal nature of FewMNER. Besides, different modal- ity models introduce bias (i.e., different similarity score distributions for text and image (Peng et al., 2018)). To this end, we propose an efficient se- lection method based on text and image similarity ranks, which can mitigate the bias described above.
3.3.3 Sorting Based on Both Similarity Ranks According to the similarity rank results of image and text modalities R. and R we sum two rank- ings and sort them to get the final ranking result.
R Rank + Re),
(10)
where R E RR and R E [I, Compared with sorting based on the sum of image and text similar- ity scores, sorting based on both similarity ranks considers the bias introduced by different modality pre-trained models. Through analyzing the distribu- tion of image similarity scores S, and text similar- ity scores S, we observe that the image similarity scores are generally higher than the text similarity scores. Sorting based on both similarity ranks can effectively address this issue. Finally, we take top- ts examples with the highest similarity ranking as selected examples.
3.3.1 Image Similarity Rank
Given an test image Vtest and candidate set DD where D contains N text-image pair and {(1,,t), (12,, t2). (uN, ty) We first adopt the pre-trained vision model VIT (Dosovitskiy et al., 2021) to obtain the representation of the whole im- age, including test image Uteat and candidate image set D.- = {11, 22,,.., ..., :::}:
H. = ViT(etest);
(2)
V = ViT(D,),
(3)
Top-K(R),
(11)
where He E Rdh is the image representation of Utest and V R-Nxd is the embedding matrix of D. Then, we calculate the cosine similarity of the test image representation H. and the image representation of the whole candidate set V, and record the rank of each candidate image set D..
where are the indices of top-k similarity ranking, and [kk
3.4 Demonstration Designing Module
Following the in-context learning paradigm (Dong et al., 2023), it consists of two parts: instruction construction and demonstration construction.
S, = Cosine (H,, V), R2 = Rank(S.),
(4)
3.4.1 Instruction Construction
(5)
We use the definition of the MNER task as the in- struction, which helps LLM understand the current task and is shown as follows:
where S. N R. E V and R E NJ.
3.3.2 Text Similarity Rank
You are a smart and intelligent Multimodal Named Entity Recognition (MNER) system. I will provide you the definition of the entities you need to extract, the sentence from where your extract the entities, the image description from image associated with sentence and the output format with examples.
Given a test text fiests we first utilize the pre- trained language model such as MiniLM (Wang et al., 2020) as text extractor to map text frest and candidate text set D,- = (41, 12, 1,,,.. I..] into dw- dimensional embedding:
To provide more detailed information for LLM, we describe the meaning of each entity category
H. MIniLM((test),,
(6)
T MiniLM(D:),
(7)
follows:
IPERSON: Short name or full name of a per- son from any geographic region; 2.ORGANI- ZATION: An organized group of people with a particular purpose, such as a business or a gov- ernment department; 3.LOCATION: Names of any geographic location, like cities, countries, continents, districts, etc; 4.MISCELLANEEUU, Name entities that do not belong to the previous three groups PERSON. ORGANIZATION, and LOCATION.
where H, E Rda is the sentence representation of frest and T E Nxdw is the embedding matrix of D.. Then, we calculate the cosine similarity of the test text representation H, and the text representa- tion of the whole candidate set T, and record the rank of cach candidate text set Dr.
S: = Cosine HH, T), R₂ = Rank(S,),
(8)
Finally, we concatenate the task and entity cate- (9) gory definitions as instruction I.
where S, E R and R2 E IR and R E NJ.
= (task definition, category definition). (12)

---

3.4.2 Demonstration Construction
Entity Type
Twitter 2015
Twiter2017
Train
As shown in the demonstration designing module in Figure 2, the demonstration template contains image description, sentence, and output. To obtain the image description, we employ the OFA model (Wang et al., 2022a) to convert images into text captions. The sentence the original text input. The output is initially constructed by concatenating the entity and the category, taking the test sample in Figure 2 as example, the initial output is "World Cup is miscellaneous." However, this leads disordered outputs, such as predicting categories that are not among the four predefined ones, despite the instruction specifying them. To address this issue, we adopt a dictionary-based for- mat that explicitly defines the output structure as ("PER": 11. "ORG": ΠT. "LOC": пI. "MISC": []]. We find that this approach effectively standardizes the output format'.
Person
2,217
552
1,816
13623
626
622
Location
731
Organization
928
247
839
1,674
375
395
Misecllancous
225
726
150
Total
6,167
1,546
5,078
6,099
1,324
1,351
Num Tweets
4,000
1,000
3,257
3,373
723
723
Table I: Statistics of two MNER datasets.
4.2 Experimental Settings
We randomly select 10, 50, 100, and all examples from the training set of two MNER datasets, respec- tively, and denote them Do.. Ds0. D100. and D Pall sets. In this paper, compare fine-tuning and few-shot methods with framework. For fine- ane-ttning methods, we utilize examples of the en- tire set to train models. For few-shot methods, we apply examples of the entire set as the train support and query sets and construct 4-way k-shot setting for training. Here, the support and query sets of few-shot baselines used for training and inference have the same entity categories. For our frame- work, select the k examples from the entire set to perform the k-shot MNER task without training.
Finally, the top-k selected examples fed into the demonstration template in ascending order such that the most similar example is nearest to the cur- rent test sample.
D = (s(D,,,,)) D
(P)) (13)
4.3 Model Settings
For the feature extraction model, employ clip- vit-base-patch322 and all-Minii.M-L6-v22 to em- bed each image and text as 112-dimensional and 768-dimensional embedding, respectively. For the image caption process, we employ the ofa-image- model to generate image description. For LLM, we choose the gpt-3.5-turbo (i.e., ChatGPT) as the backbone of our framework.
3.5 Predict module
Given the instruction and demonstration, we con- catenate them into the whole context C. Then, we feed context C' and test sample (Ptest: ftest) into LLM and select the most probable generated se- quence the predicted output.
C (I,
(14)
1 max PECM(Y) C,seatett,tae treat)).
(15)
4.4 Comparison Models
Finally, we decode the prediction output into a list according to the dictionary format and complete the k-shot MNER task.
For fine-tuning methods, we adopt the following baselines: (1) UMT (Yu et al., 2020), which em- ploys transformer layer with multimodal in- teraction module capture the inter-modality dy- namics between tokens and images for MNER; (2) UMGF (Zhang et al., 2021), which applies a unified multimodal graph approach to capture semantic re- lationships between tokens and visual objects and performs entity labeling; (3) HVPNeT (Chen et al.. 2022b), which utilizes a hierarchical visual prefix fusion network for the visual-enhanced entity; (4)
4 Experiment
4.1 Dataset
We conduct experiments modal named entity recognition (MNER) bench- mark datasets, Twitter2015 and Twitter2017. Two MNER datasets are constructed by (Yu et al., 2020). Each example consists of a text and an associated image in the two MNER datasets. The statistics of two MNER datasets are shown in Table 1.
public multi-
hnttps://huginggfaat http://uugggngaaaccccc/eeeeeeat hip-vittbbae-paathht
'nttps://huggingfaat http:///gggnnaaaacec//////nnnnnt noossetenneetransforooee
'To support this statement, we perform experiments com- paring the two diffcrent output formats (detailed results in Appendix A.
all-MiniLN-L6-v2
"https:
http://oodllllcccc http:ssccpppcccceeeeeeeeeeeeeeee http:///////////m//o/ooraaa/e

---

D10
D50
D100
Daul
Setting
Method
P(%)
R(%)
F1(%)
P(%)
R(%)
FI(%)
P(%)
R(%)
FI(%)
P(%)
R(56)
F1(%)
Twitter2015
UMT
1.47
0.54
0.79
41.10
39.12
40.09
48.63
57.42
52.66
71.67
75.23
73.41
UMGF
0.95
2.29
1.34
45.84
41.08
43.33
48.18
56.15
51,86
74.49
75.21
74.85
FT
HVPNET
29.55
15.31
20.17
56.12
42.64
48.46
59.96
56.06
57.94
73.87
76.82
75.32
DehiasCI.
0.98
7.65
1.74
43,97
36.54
39.91
51.30
49.60
50.44
74.45
76.13
75.28
ProtoBERT
32.35
51.45
41.20
35.67
53,36
42.75
2-shot
StructShot
35,61
40.51
37.89
31:40
39,42
34.94
Ours
45.03
63.72
52.76
46.39
63.82
53.73
45.96
64,95
53.83
47.73
65.25
55.13
ProtoBERT
32.40
53.04
40.23
35.63
56.34
43.64
4-shot
StructShot
30.26
37.20
33.34
26.66
35.13
30.14
Ours
47.36
65.01
54,80
48.87
65.87
56.11
47.5%
65.73
55,20
50.21
68.28
57.86
ProtoBERT
32.31
54.75
40.62
35.38
58.37
44.04
8-shot
StructShot
32.39
39.78
35.70
24.81
32.04
27.78
Ours
50.44
64.39
56.57
49.94
64.13
56.15
48.72
64.35
55.45
51.24
67.20
58.14
Twitter2017
UMT
5.13
3.03
3.81
56.14
58. 18
57.14
62.70
65.58
64. 11
85.28
85.34
85.31
UMGF
1.41
2,00
1.65
58.30
54.81
56.50
65.09
64.94
65.01
86,54
84.50
85.51
FT
HVPNeT
35.44
31.90
33.58
61.56
55.96
58.63
66.21
64.10
65.14
85.84
87.93
86.87
DebiasCL
1.93
12.85
3.36
50.40
35.20
41.45
70.48
65,36
67.83
87.59
86.11
86.84
ProtoBERT
42.51
58.99
49.39
48.91
63.58
55,28
2-shot
StruciShol
41,51
50.68
45.63
49.66
57.12
53.12
Ours
61.30
72.09
66.26
62.40
71.72
66.74
64.79
73,43
68.84
65.82
74.54
69.91
ProtoBERT
42.50
61.56
50.27
49.32
66.37
56.58
4-shot
StructShox
40.40
50.47
44.72
47.26
56.26
51.35
Ours
64.08
74.46
68.88
65.58
74.46
69.74
66.05
76.46
70.87
67.27
77.28
71.92
ProtoBERT
37.35
59.75
49.67
50.48
66.73
57.47
8-shot
StructShot
42.09
53.17
46.98
47.66
55.13
51,10
Ours
65.33
73.35
69.11
66.76
73.58
70,00
69.24
77.13
72.97
69.00
78.09
73.26
Table 2:
Main experiment results to compare fine- teeinunngg and few-shot baselines with our framework in 2-shot, 4-shot, and 8-shot settings on D 10. D50. D100 and au sets of Twitter2015 and Twitter2017 datasets. FT denotes the fine-tuning method. For Do set, results with due to 4-way 2/4/8-shot setting more than number of D10 set. For Dau set, few-shot methods are similar to fine-tuning methods. Therefore, we use "-" to indicate that this is not valid few-shot setting. To ensure the reliability of few-shot baselines, the results of ProtoBERT and StruetShot are the average results of 100 runs.
DebiasCL (Zhang et al., 2023), which proposes a diabias contrastive learning- beaning-basedd based approach for MNER and studies modality alignment enhanced by cross- modall contrastive lcarning.
our framework effectively exploits the in-context learning potential of large language models. (2) Compared with few-shot baselines such as ProtoBERT and StructShot, our framework achieves the best results in all few-shot settings (i.e., 2-shot, 4-shot, and 8-shot). This indicates that methods based on in-context learning are prefer- able in the FewMNER task, and thus exploring congenial methods based on in-context learning can lead improved performance in this task.
For the few-shot methods, we apply the follow- ing baselines: (1) ProtoBERT (Ding et al., 2021), which employs prototypical network with a back- bone of BERT encoder; (2) StructShot (Yang and Katiyar, 2020), which uses token-level nearest neighbor classification and structured inference.
(3) We observe that the performance of our framework improves as the size of D increases. This is because a larger retrieval set provides more opportunitics for the test sample to find similar examples.
4.5 Main Results
We report the main experimental results Table and draw the following conclusions.
(1) Our framework significantly outperforms the fine-tuning methods on 10. D50 and D 100 sets (except for HVPNeT D100 set of Twitter2015). For example, in terms of F1, our framework outperforms UMT by 55.78% and 65.30%, UMGF by 55.23% and 67.46%, HVPNeT by 36.40% and 35.53%, and DebiasCL by 54.83% and 65.75% on Do0 set of two MNER datasets. These show that
(4) Our framework still lags behind the fine- tuning methods under the D all set.
4.6 Ablation Study
To analyze the impact of instruction and demon- stration on the performance our framework, we conduct ablation experiments and report detailed

---

Twirter2015
Twiner2017
Methods
2 shot
4-sbot
2-shot
4 shshot
PC26)
R(76)
F1(76)
P176)
R(7))
F1(%)
P(7))
RC%)
F1(%)
P(2)
R(%)
F11(6)
46.39
63,82
66.74
69.74
wif
45,46
63.17
52,88
48.50
64.33
55.30
60.55
71.58
50.60
65.39
73.58
69.24
wofo D
43.35
40.01
41.61
43.35
40.01
41.61
63.44
$3.29
$7.92
63.44
53.29
57.92
w/ score
46.47
63.68
53.73
46.78
64.07
54.08
62.55
70.10
66.11
64.42
69.95
67.07
Table 3: Ablation study in 2-shot and 4-shot settings on Dno set. I and D denote instruction and demonstration,
respectively. w/ score indicates that examples are selected based on the sum of image and text similarity scores.
Twitter2015
Twitter2017
Modality
Single Type (H1)
Overall
Single
Type (F1)
Overall
PER
LOC
ORG
MISC
P(%)
R(%)
(T(%)
PER
1.C
ORG
MISC
[(7)
R(%)
F1(%5
single
67.08
39.86
17.48
88.96
72.34
18.18
64,29
68.98
malri
77.33
65.91
38.54
20.21
48.87
65.87
56.11
89.50
71.26
66.23
17.33
65.58
74.46
69.74
Table 4: Different modalities 4-shot setting on Doo set.
TWi5er2015
Twitter2017
Twiner2015
Twitter2017
Methods
Methods
P(56)
R(%)
FI(56)
P(7)
R(%)
HI(7))
P(%)
R(%)
F1(5)
P(56)
R(%)
FT(%)
madom
47.46
64.54
54.70
64.21
73.58
68.58
descending
48.61
65.29
58.73
65.14
74.98
69.71
dissimilar
47.08
64,56
54.46
61.84
71.13
66.
random
48.43
65.42
58.66
65.49
74.32
69.62
similar
48.87
65.87
$6.11
65.58
74.46
69.74
ascending
48.87
65.87
56.11
65.58
74.46
69.74
Table 5: Different selecting example methods in 4-shot setting on D50 set.
Table 6: Different sorting example methods in 4-shot
setting on 50 set.
Different Examples Analysis. To analyze the impact of different examples, we compare three methods of examples selection (ie., similar, dis- similar, and random) in the 4-shot setting on Do0 set. The similar method selects examples that have the highest similarity to the test sample, while the dissimilar method seleets examples that have the lowest similarity. The random method selects ex- amples uniformly at random. The results are shown in Table 5. We observe that the similar method achieves the best performance, followed by the ran- dom method, and the dissimilar method performs the worst. This indicates that selecting similar ex- amples to form the demonstration beneficial for the FewMNER task.
results in Table 3. The results reveal that our frame- work achieves the best performance when combin- ing instruction and demonstration, which suggests that both components are beneficial. Compared with removing instruction, removing demonstration leads to more performance degradation. This shows that the demonstration is crucial for our framework. Furthermore, we also conduct ablation experi- ments on the way of selecting examples. Sorting based on the sum of image and text similarity ranks outperforms sorting based on the sum of image and text similarity scores (i.e., w/ score). This is because the latter does not account for the bias introduced by different modality models.
4.7 Analysis
Impact of Examples Sort. To investigate the im- pact of example sort on performance, we utilize the same examples to compare three methods of sort- ing (i.e., ascending, descending, and random) in the 4-shot setting on 250 set. The results are shown in Table 6. The ascending sort method, which places the most similar example nearest to the current test sample, outperforms the other methods. This suggests that ascending sort examples by their sim- ilarity can improve performance. The reason is that the most similar example can provide more relevant information for the current prediction.
Image Modality Analysis. To explore the ef- feet of image description on the FewMNER task, we conduct experiments with single-modality (i.e..
text) and multi multi-modality (i.e., text and image) and show results in Table 4. For a fair comparison, both settings use the same instruction and demonstra- tion, but the single-modality setting discards the image description. We observe that the multi- modality setting outperforms the single-modality setting, especially on the PER category. The reason is that the image caption model tends to generate sentences related to people, which provide useful cues for identifying PER category entities.
Impact of the Number Examples. To explore the impact of the number of examples on perfor-

---

PER
LOC
ORG
MISC
Overall
Settin
P(%)
R(%)
F1(%)
PC963
R(%)
FI(S))
PCS
R(%)
F1(%)
P(S))
RC%)
F1(%0
F1(30
2015
2-shot
74,90
66.37
64.99
24.96
61.03
35.43
24.86
53.73
4.shot
72.67
82,63
77.33
66.67
68,16
65.91
27.30
65.55
38.54
16.65
25.69
20.21
56.11
8-sbot
72.14
80.71
76.18
66.99
64,70
65.82
27.79
63.53
38.67
16.68
21.89
18.86
56.15
7wi1ter2017
2-shot 90.75
87.10
69.66
69.66
56.87
74.43
10.88
19.75
4 4-shol
92.01
87.12
89,50
72.94
69.66
71.26
57.84
77.47
66.23
14.17
22.29
17.33
69.74
8-shot 92.44
84.70
88.40
70.17
71.35
70.75
$9.53
77.47
67.83
15.56
22.29
18.32
70.00
Table 7: The detail metric (i.., P. R. and F1) for four categories in 2-shot, 4-shot, and 8-shot settings D50 set.
Twirte22015
Twirter2017
Setting Np
B,
Na
C.
FI(SST
N₂
B3%%
Na.
C.
FI(9E)
7038
45.15
15.41
1553
28.59
1109
12.62
4-shol
6896
42.92
3936
14.38
56.11
1534
25.55
1142
11.91
69.74
8-sbot
6570
41.35
3853
14.85
56.15
1489
23,50
1139
12.73
70.00
Table 8: Analysis of the wrong predictions in 2-shot, 4-shot, and 8-shot settings on D50 set. Np. B., Neb. and C, denote the number of predictions, proportion of boundary errors, correct number of boundaries, and proportion of category errors on the FewMNER task, respectively.
Serting
Twitte20015
Twitter2017
the performance of four categories in 2-shot, 4- shot, and 8-shot settings on D50 set. We find that our framework performs poorly on MISC category, which is significantly lower than PER, LOC, and ORG categories. The reason is that MISC miscella- neous category. defined as name entities that do not belong to the previous three categories. The annotation of MISC category entities depends on the preference of annotators. Relying only on the in-context learning ability of LLM and a few exam- ples is not sufficient to learn this preference.
P(%)
R(%)
F1(%)
P(
R(%)
shot
43.35
40.01
41.61
63.44
53.29
57.92
shot
46.39
63.82
53.73
62.40
71.72
66.74
4-shot
47
65.87
56.11
65.58
74.46
68.74
8-shot
49.94
64.13
56.15
66.76
73.58
70.00
16 shot
50.83
64.86
$6.99
68.04
74.69
71.21
32-shot
49.35
64.50
55.92
69.31
74.38
71.76
Table 9: Different shot settings on D50 set.
mance, we conduct experiments with different the number of examples (i.e., 0, 2, 4, 8, 16, 32-shot) on D0 set and show results in Table 9, On the Twitter2017 dataset, we observe that the Fl gener- ally increases with the number of examples, and our framework achieves the best score with 32 ex- amples. Comparing 0-shot with 32-shot, the latter outperforms the former by 13.84% in Fl.
Moreover, we analyze the boundary error and category error and perform a detailed analysis of wrong predictions. We classify wrong predictions into two types: boundary errors and category er- rors We count the number of errors for each cate- gory and report results in Table 8. We observe that increasing the number of examples significantly reduces boundary errors. Specifically, comparing 2-shot with 8-shot, the latter reduces the proportion of boundary errors by 3.80% and 5.09% on two datasets, respectively. Besides, increasing the num- ber of examples does not reduce category errors. This is an interesting finding and demonstrates that more examples mainly improve the boundary abil- ity of ICL, rather than category ability.
Recently, some works have attempted to explain the ICL capability of LLMs. Dai et al. (2023) in- terprets language models as meta-optimizers and views ICL as a form of implicit fine-tuning. This is consistent with our findings that more examples can enhance the performance, as more examples lead to more optimization steps. On the Twitter2015 dataset, we observe similar trend the Twit- ter2017 dataset, but framework achieves the best score with 16 examples. The reason is that increasing the number of examples may introduce more dissimilar examples. These fluctuations indi- cate that more examples can have a positive effect on performance if they are sufficiently similar.
Conclusion
In this paper, we formulate multimodal named en- tity recognition as a few-shot learning problem,
Error analysis. In this section, we aim to an- alyze the factors that affect the performance of our framework. As shown in Table 7, we report
'When the predicted entity is coundary-mispeecinng classify it as boundary error. When the boundary of the entity is completely correct, but the category is incorrectly identified, we classify it as a category error.
we

---

named few-shot multimodal named entity recog- nition (FewMNER), to extend entity detection to unseen entity categories. To tackle FewMNER, we propose a framework based on in-context learning by addressing three problems. Experimental results show that our framework outperforms baselines in several few-shot settings. Moreover, we conduct analysis experiments and find that selecting simi- lar examples, sorting them in ascending order, and using more examples improve the performance of in-context learning. We also perform error analysis and observe that increasing the number of exam- ples reduces boundary errors but not category er- rors. These results provide novel insights for future work on the FewMNER task.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NIPS, 13:1877-1901.
Hailin Chen, Amrita Saha, Shafiq Joty, and Steven C.H. Hoi. 2022a. Learning label modular prompts for text classification in the wild. In Proceedings of EMNLP, pages 1677-1690.
Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. 2021. Can images help recognize entities? study of the role of images for multimodal ner. In Proceedings of W-NUT, pages 87-96.
Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao. Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022b. Good visual guidance make better extractor: Hierarchical visual prefix for multi- modal entity and relation extraction. In Findings of NAACL, pages 1607-1618.
Limitations
Although the proposed framework significantly out- performs several strong baselines on the FewM-
NER task, it suffers from the following limitations:
Damai Dai, Yutao Sun, Dong. Yaru Hao, Shuming Ma. Zhifang Sui, and Furu Wei. 2023. Why can GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers. In Proceedings of MT-FoMo.
In the case of using the full amount of data, our framework still lags behind fine-tuning methods. We can utilize some data to domain fine-tune LLM before applying our frame- work, which may further improve the perfor- mance of in-context learning on the FewM- NER task. This is a direction for future ef- forts.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of NAACL, pages 4171- 4186.
Ning Ding. Guangwei Xu, Yulin Chen, Xiaobin Wang. Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. In Proceedings of ACL, pages 3198-3213.
Unfortunately, due to API limitations, we are unable to obtain results from the more pow- erful GPT4 model, which has a multimodal function. Further experiments and analysis are required.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey in-context learning. arXiv preprint arXiv:2301.0025
We believe that addressing the above limitations can further improve the performance of our frame-
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of ICLR.
work.
Acknowledgements
This work was partially supported by the Na- tional Natural Science Foundation of China 62006062 and 62176076), Natural Science Foundation of GuangDong 2023A1515012922. Shenzhen Foundational Research Funding Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies 2022B 12000000
Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named entity recognition task. In Proceedings of SAC, pages 993- 1000.
(CYI20200109113444
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- context learning for few-shot dialogue state tracking. In Findings of EMNLP, pages 2627-2643.

---

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of DeeLIO, pages 100-114.
Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal trans- former. In Proceedings of ACL, pages 3342-3352.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging multimodal social media. In Proceedings of ACI. pages 1990-1999.
Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multi- modal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of AAAI, pages 14347-14355.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of EMNLP, pages 11048-1100 11064.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named en- tity recognition in tweets. In Proceedings of AAAI, pages 5674-5681.
Xin Zhang, Jingling Yuan, Lin Li, and Jianquan Liu. 2023. Reducing the bias of visual objects in multi- modal named entity recognition. In Proceedings of WSDM, page 958-966.
Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. Proceedings of NAACL, pages 852-860.
Yiming Zhang. Shi Feng, and Chenhao Tan. 2022. Ac- tive example selection for in-context learning. In Proceedings of EMNLP. pages 9134-9148.
Yuxin Peng, Jinwei Qi. and Yuxin Yuan. 2018. Modality-specific cross-modal similarity measure- ment with recurrent attention network. IEEE TIP. 27(11):5585-559
A Appendix
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. Proceedings of NAACL, pages 2655- 2671.
A.1
Impact of Different Output Format
Twitter-2015
Methods
P(%)
R(%)
F1(%)
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley. Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint urXiv:2201.11991
sentence
51.10
51.84
51.47
dictionary
48.87
65.87
56.11
Table 10: Impact of different output format in 4-shot setting on Dso set of Twitter-2015.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for language asdelage-mode--ss--ee model-as-a-seree seeeevicc.. In Proceedings of ICML, pages 20841-20855
Twitter-2017
Methods
P(%)
R(%)
F1(%)
sentence
61.41
61.73
61.57
dictionary
65.58
74.46
69.74
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. Ofa: Unifying architectures, tasks, and modalities through a sim- ple sequence-to-sequence learning framework. In Proceedings of JCML, pages 53318-23344
Table II: Impact of different output format in 4-shot setting on Po0 set of Twitter-2017.
To explore the influence of different output formats, we perform experiments comparing bacee output format and sentence- output format and show results in Table 10-11. We can obverse that compared with the sentence- based output format, the dictionary-based output format achieves 4.64% and 8.17% higher Fl in 4-shot setting on D50 set of Twitter2015 and Twit- ter2017 7 datasets, respectively, which demonstrates the dictionary-basee output format more suitable for in-context learning paradigm on ChaiGPT. Per- haps this is related to the pre-training of ChatGPT on code, as this code-like output format is more comprehensible for ChatGPT.
Wenhui Wang. Furu Wei, Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trainee trained transformers. NIPS, 33:5776-577 5788.
Xinyu Wang. Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, and Kewei Tu. 2022b. Ita: Image-text alignments for multi-moda named entity recognition. In Proceedings of NAACL, pages 3176-3189.
Yi Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. In Proceedings of EMNLP, pages 6365-6375.

---

Twitter2015
Twitter0117
Replace Proportion
P(%)
R(%)
F1(%)
P(%)
R(%)
F1(%)
0%
56.43
61.10
58.67
72.51
71.06
71.78
10%
56.32
60.85
58.50
70.87
70.76
70.81
50%
52.88
58.97
55.76
67.05
68.39
67.72
100%
51.39
58.54
54.73
61.65
61.88
61.77
Table 12:
Different labels replace proportion in 4-shot setting on so set. Note that the results presented here differ from the main experiment of paper due OpenAl updating ChatGPT.
A.2 Perturbation Analysis
We conduct a perturbation analysis to examine the effect of label noise in-context learning. We randomly replace different proportions of labels with other labels in the examples and measure the impact of label noise on the Fl score of our frame- work. The results are shown Table 12. As ex- pected, the F1 score decreases as the label replace- ment ratio increases. When all the example labels are replaced (i.e., 100%), the FI score drops by 3.94% and 10.01% on two datasets, respectively. However, the Fl score does not drop to zero when all the example labels are replaced, which indicates that the L.LM has a strong intrinsic abil- ity for this task and is not completely lost by the examples.
A.3 Analyze Our Framework from Task Perspective
We here provide an analysis of why our framework performs well from the perspective of the F-wM- NER task. First, model the FewMNER task generative task, which is more direct than the baseline methods that model the MNER task as sequence labeling task. Sequence labeling meth- ods not only require fine-grained prediction but also need to learn the association between differ- ent labels. Second, our framework utilizes image caption a bridge to alleviate the gap between image modality and text modality, which can skip the process of information interaction between dif- ferent modalities. Third, the four category con- cepts of people, location, organization, and miscel- laneous FewMNER tasks can be understood by LLM. Our framework can be readily adapted to the task objectives. For these reasons, our framework achieves good performance on the FewMNER task under the condition of small samples.


Reviews:
[{"review_id": "RTtMm3UjUD", "reviewer": "Reviewer_DFxJ", "summary": "", "questions": "A. Are there other metrics for calculating the Cos distance of image representations?\n\nB. What is the few-shot setting in sampling MNER instances? For example, when a sample contains multiple entities, it is unclear whether it should be considered as one sample or multiple samples in the sampling process. If it is counted as one, it is also unclear how to ensure that each entity category occurs only once in the k-shot setting.\n\nC. We know that the construction of Demonstration is very important for ICL, have the authors considered whether slight perturbations can enhance ICL or affect it, and these aspects of consideration and analysis would be interesting to see.\n\nD. How can we avoid that LLM has learned all the data in the dataset during the pretraining process, how do the authors see or avoid this problem?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you've taken to review our work.\n\n**Comment 1**\n\n>The data comparison is solely based on a single source, specifically social media datasets from Twitter. However, the results and the model's ability to generalize are not strongly supported.\n\n**Response**: \n\nThanks for this insightful comment. Indeed, using datasets from other sources to support the performance and generalizability of the proposed framework is required. However, to the best of our knowledge, there are currently only two publicly available datasets (i.e., Twitter2015 and Twitter2017) for the MNER task. This is why we only utilize these two datasets to evaluate the proposed framework in this paper. Besides, we are annotating a new MNER dataset, which covers several different social platforms. We will conduct research on this diverse dataset in the future.\n\n--------\n\n**Comment 2**\n\n>The proposed method is primarily based on LLM and ICL. However, it heavily depends on the accuracy of image-caption in the detailed processing, which can lead to the problem of error propagation\n\n**Response**: \n\nThanks a lot for this interesting comment. We agree with your viewpoint that if the generated caption for each image is not quite accurate, the image-caption module would cause errors to propagate through our framework. In preliminary experiments, we randomly selected 100 samples to evaluate the quality of image captions. We found that most of captions are accurate and factual, so this error is small.\n\n------\n\n**Comment 3**\n\n>Although the primary focus of the paper is on ICL, the essential procedures involved, such as retrieve-based methods and similarity rank, are general in V+L models. However, the ablation experiments fail to distinguish the contribution of these aspects, which undermines the overall credibility of the main innovation presented in the paper.\n\n**Response**: \n\nWe apologize for not making it clear. To distinguish the contribution of the retrieve-based methods proposed in this work, we compare the similar, dissimilar, and random retrieve methods and conduct experiments in sub-section Different Examples Analysis (see Table 5). The results show that the similar method achieves the best performance, followed by the random method, and the dissimilar method performs the worst. \n\nFor similarity rank, we present results in the ablation study (see w/ score in Table 3). Here, w/ score indicates that examples are selected based on the sum of image and text similarity scores. From this table, we can see that sorting based on the sum of image and text similarity ranks outperforms that of image and text similarity scores (i.e., w/ score). \n\nWe hope that these experimental comparisons can distinguish the contribution of these aspects. To make it more convincing and clearer, we will amend the relevant text description.\n\n-------\n\n**Comment 4**\n\n>The main focus in this paper is not clearly addressed when it comes to few-shot learning. There have been previous studies on few-shot named entity recognition (FNER), but this paper is confusing because it does not provide clear guidance on how to sample in few-shot MNER.\n\n**Response**: \n\nVery sorry for the unclear expression. To answer your confusion, we here provide clearer guidance on how to sample in a few-shot MNER. Different from the previous few-shot NER, which constructs an N-way K-shot setting, our proposed FewMNER task utilizes only K-shot, i.e., selects K samples from the training dataset and does not consider whether each entity category occurs N times.  To illustrate this sample, we provide an example in the Introduction. As shown in Figure 1(b), the 2-shot MNER task aims to accomplish MNER based on two labeled text-image pair examples. We hope that these explanations will give you a clearer understanding of how to sample in few-shot MNER.\n\n---------\n\n**[Q1]** Are there other metrics for calculating the Cos distance of image representations?\n\n**[A1]** Yes, there are other metrics such as L1, L2 and Dot Product. We compare their performance on the FewMNER task. The experimental results are as follows:\n\n| Distance  Metrics | Twitter2015(F1%) | Twitter2017(F1%) |\n| ----------------- | :--------------: | :--------------: |\n| L1                |      59.16       |      71.90       |\n| L2                |      58.79       |      71.94       |\n| Dot Product       |      58.59       |      70.77       |\n| Cosine            |      58.67       |      71.78       |\n\n*Table 1: Different distance metrics in 4-shot setting on $\\mathbf{\\mathcal{D}_{50}}$ set. Note that the results presented here differ from paper due to OpenAI updating ChatGPT.*\n\nFrom the results, we can notice that the results of different metrics are similar, indicating that the effect of distance metrics is small. It is fine to select any reasonable distance metric. In our work, we originally chose cosine similarity because it is commonly used.\n\n**[Q2]** What is the few-shot setting in sampling MNER instances?\n\n**[A2]** This question is similar to **Comment 4**. We view one sample as 1-shot. For example, 2-shot utilizes only two labeled text-image pair examples. Different from the previous few-shot NER task that constructs N-way K-shot setting, our proposed FewMNER task utilizes only K-shot, i.e., selects K samples from the train dataset, and does not consider whether each entity category occurs N times. Due to the ICL paradigm, we do not need to construct N-way K-shot setting to ensure that each entity category appears only once. It is worth noting that few-shot baselines in the main experiment keep N-way K-shot setting since they apply meta-learning for training.\n\n**[Q3]** We know that the construction of Demonstration is very important for ICL, have the authors considered whether slight perturbations can enhance ICL or affect it, and these aspects of consideration and analysis would be interesting to see.\n\n**[A3]** Thanks a lot for your very interesting question. In this paper, we conduct experiments to explore the impact of example sort (see Table 6). We use the same examples and apply different sorting methods, i.e., ascending, descending, and random. The results show that the ascending sort method slightly outperforms the other methods. This suggests that different orderings of the same examples have little effect on ICL. \n\nThanks for the reminder, we perform another perturbation experiment. We randomly replace different proportions of labels with other labels to explore the impact of misinformation on ICL. The experimental results are as follows:\n\n| Replace Proportion | Twitter2015(F1%) | Twitter2017(F1%) |\n| ------------------ | :--------------: | :--------------: |\n| 0%                 |      58.67       |      71.78       |\n| 10%                |      58.50       |      70.81       |\n| 50%                |      55.76       |      67.72       |\n| 100%               |      54.73       |      61.77       |\n\n*Table 2: Different labels replace proportion in 4-shot setting on $\\mathbf{\\mathcal{D}_{50}}$ set. Note that the results presented here differ from paper due to OpenAI updating ChatGPT.*\n\nAs the label replacement ratio increases, F1 will continue to decline. Comparing replacing all example labels (100%) with not replacing examples (0%), the former reduces the F1 by 3.94% (58.67%->54.73%) and 10.01% (71.78%->61.77%) on two datasets, respectively. Furthermore, we can notice that the F1 score does not drop to 0 when replacing all example labels (100%), which shows that the LLM has strong self-ability for this task and is not completely lost by the examples. This perturbation experiment is very interesting, thanks again for your suggestion. We will supplement this part of the experiment in the next edition of the paper.\n\n**[Q4]** How can we avoid that LLM has learned all the data in the dataset during the pretraining process, how do the authors see or avoid this problem?\n\n**[A4]** Thank you for your insightful question and your interest in our work. This is a very interesting question that got me thinking. Here, We share our viewpoint. We can simply split the ability of LLM into two aspects: self-ability and incremental ability. Self-ability can be viewed as the performance of LLM on zero-shot learning, which does not use any examples from the target domain. Incremental ability can be viewed as an improvement of LLM on K-shot learning, which uses a few examples from the target domain as demonstration.\n\nWe hypothesize that if LLM has learned all the data in the dataset during the pretraining process, then adding more examples from the same dataset would not have much impact on its performance. However, as shown in Table 9, we observe that the F1 score generally increases with the number of examples on both Twitter2015 and Twitter2017 datasets. For example, comparing 0-shot and 32-shot, the F1 scores are 41.61% vs. 55.92% and 57.92% vs. 71.76% on the two datasets, respectively. According to the above analysis, I tend that LLM has not learned all the data in the dataset during the pretraining process and can still benefit from incremental learning with demonstration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Currently, the FewMNER problem receives not much attention, however, it remains significant in the domains of multimodal and NER processing.\n    \n- The paper presents a potentially promising approach for addressing the FewMNER task through In Context Learning and LLM usage from a limited number of examples without the need for training.\n\n- The comparative experiments are adequate, and the results also show that the method proposed in the paper is competitive.", "reasons_to_reject": "- The data comparison is solely based on a single source, specifically social media datasets from Twitter. However, the results and the model's ability to generalize are not strongly supported.\n\n- The proposed method is primarily based on LLM and ICL. However, it heavily depends on the accuracy of image-caption in the detailed processing, which can lead to the problem of error propagation.\n\n- Although the primary focus of the paper is on ICL, the essential procedures involved, such as retrieve-based methods and similarity rank, are general in V+L models. However, the ablation experiments fail to distinguish the contribution of these aspects, which undermines the overall credibility of the main innovation presented in the paper.\n- The main focus in this paper is not clearly addressed when it comes to few-shot learning. There have been previous studies on few-shot named entity recognition (FNER), but this paper is confusing because it does not provide clear guidance on how to sample in few-shot MNER. \n\n", "questions_to_authors": "A. Are there other metrics for calculating the Cos distance of image representations?\n\nB. What is the few-shot setting in sampling MNER instances? For example, when a sample contains multiple entities, it is unclear whether it should be considered as one sample or multiple samples in the sampling process. If it is counted as one, it is also unclear how to ensure that each entity category occurs only once in the k-shot setting.\n\nC. We know that the construction of Demonstration is very important for ICL, have the authors considered whether slight perturbations can enhance ICL or affect it, and these aspects of consideration and analysis would be interesting to see.\n\nD. How can we avoid that LLM has learned all the data in the dataset during the pretraining process, how do the authors see or avoid this problem?\n", "paper_topic_and_main_contributions": "The main focus of the paper is on the few-shot Multimodal Named Entity Recognition (FewMNER) task, specifically, locating and identifying named entities for a text-image pair only using a small number of labeled examples. This paper proposes an in-context learning paradigm to leverage a large language model to solve FewMNER. Especially, for three potential problems which may hinder the ICL paradigm, this paper first leverages the image caption model to generate textual description; second, they construct a similarity rank for multimodal; and they add the description of entity category meaning to the instruction and feed the concatenation of instruction, demonstration and test sample to LLM for predicting the entity tags. The main contribution of the article is to provide a solution paradigm for MNER in a few-shotscenario.\nAround this In-context learning paradigm, the authors' study provides certain insights.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kigplC3kZY", "reviewer": "Reviewer_BB5Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you've taken to review our work.\n\n**Comment 1**\n\n>This work appears to lean more towards an applied paper rather than an academic research paper. It extensively relies on existing techniques (e.g., Image caption model and ChatGPT), with limited original contributions.\n\n**Response**: \n\nThanks for your comment. Although our framework leverages existing techniques (e.g., Image caption model and ChatGPT), we maintain that our work has academic value. The specific reasons are as follows:\n\n- Many studies have explored the in-context learning (ICL) of LLM, but they mostly focus on generation and classification problems. Solving extraction tasks in multimodal scenarios is still a challenge. In this paper, our main contribution is to validate the feasibility of ICL for FewMNER task and analyze the impact of ICL on the performance of FewMNER task. \n- For the framework, we design simple and effective modules (e.g., retrieve-based method, similarity rank, output format) according to multimodal and extraction task characteristics, which have a good generalization and can be easily applied to other multimodal extraction tasks (e.g., multimodal event extraction).\n- Besides the framework, we focus on the demonstration designing module and conduct extensive experiments to obtain interesting findings, e.g., selecting similar examples, sorting them in ascending order, and using more examples can improve the performance of ICL on FewMNER task. Furthermore, we perform error analysis and observe that increasing the number of examples reduces boundary errors but not category errors. These findings can provide novel insights and guidance for feature work on FewMNER and ICL. \n\nWe hope these expositions can demonstrate the academic value of our work.\n\n-------\n\n**Comment 2**\n\n>While the experimental results are promising, the credit for the success largely goes to the Large Language Model (LLM) employed. The paper, however, lacks an in-depth analysis from the perspective of MNER to explain why the performance is good, resulting in limited interpretability.\n\n**Response**: \n\nSorry for the lack of this analysis. To answer your concerns, we here provide an analysis on why our framework performs well from the perspective of FewMNER task. First, we model the FewMNER task as a generative task, which is more direct than the baseline methods that model the MNER task as a sequence labeling task. Sequence labeling methods not only require fine-grained prediction but also need to learn the association between different labels. Second, we utilize image caption as a bridge to alleviate the gap between image modality and text modality, which can skip the process of information interaction between different modalities. Third, the four category concepts of people, location, organization, and miscellaneous in FewMNER tasks can be understood by LLM. Our framework can be readily adapted to the task objectives. For these reasons, our framework achieves good performance on the FewMNER task under the condition of small samples. We appreciate your valuable suggestion and we will include this analysis in the next version of the paper.\n\n-------\n\n**Comment 3**\n\n>Some recent relevant works have not been compared, such as: 1). M3S: Scene Graph Driven Multi-Granularity Multi-Task Learning for Multi-Modal NER; 2). Reducing the Bias of Visual Objects in Multimodal Named Entity Recognition.\n\n**Response:**\n\nThank you for your thoughtful reminder of these two articles. Unfortunately, we could not find the open source for the paper 1). M3S: Scene Graph Driven Multi-Granularity Multi-Task Learning for Multi-Modal NER. We are trying to reproduce this paper ourselves, but we have not completed it yet due to time constraints. Luckily, we obtain the code for the paper 2). Reducing the Bias of Visual Objects in Multimodal Named Entity Recognition, namely DebiasCL. The experimental results are as follows. We can observe that our framework still significantly outperforms the DebiasCL methods on $\\mathbf{\\mathcal{D} _{10}}$, $\\mathbf{\\mathcal{D} _{50}}$, and $\\mathbf{\\mathcal{D} _{100}}$ sets. \n\n\n| Setting | Method   | $\\mathbf{\\mathcal{D}_{10}}$(F1%) | $\\mathbf{\\mathcal{D}_{50}}$(F1%) | $\\mathbf{\\mathcal{D}_{100}}$(F1%) | $\\mathbf{\\mathcal{D}_{all}}$(F1%) |\n| ------- | -------- | :------------------------------: | :------------------------------: | :-------------------------------: | :-------------------------------: |\n|       | UMT      |               0.79               |              40.09               |               52.66               |               73.41               |\n|         | UMGF     |               1.34               |              43.33               |               51.86               |               74.85               |\n|         | HVPNeT   |              20.17               |              48.46               |               57.94               |               75.32               |\n| FT        | DebiasCL |               1.74               |              39.91               |               50.44               |               75.28               |\n| 8-shot  | **Ours** |              56.57               |              56.15               |               55.45               |               58.14               |\n\n*Table 1: The experiment results on Twitter2015.*\n\n| Setting | Method   | $\\mathbf{\\mathcal{D}_{10}}$(F1%) | $\\mathbf{\\mathcal{D}_{50}}$(F1%) | $\\mathbf{\\mathcal{D}_{100}}$(F1%) | $\\mathbf{\\mathcal{D}_{all}}$(F1%) |\n| ------- | -------- | :------------------------------: | :------------------------------: | :-------------------------------: | :-------------------------------: |\n|       | UMT      |               3.81               |              57.14               |               64.11               |               85.31               |\n|         | UMGF     |               1.65               |              56.50               |               65.01               |               85.51               |\n|         | HVPNeT   |              33.58               |              58.63               |               65.14               |               86.87               |\n|  FT       | DebiasCL |               3.36               |              41.45               |               67.83               |               86.84               |\n| 8-shot  | **Ours** |              69.11               |              70.00               |               72.97               |               73.26               |\n\n*Table 2: The experiment results on Twitter2017.*\n\nWe will continue to reproduce the code of the first paper (i.e., M3S) and supplement this comparison result in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is the first attempt to combine few-shot learning with multimodal named entity recognition (NER), introducing the FewMNER task, and this work is also the first exploration of the potential of in-context learning in FewMNER.\n2. This work conducts comprehensive experiments to validate the effectiveness of the proposed algorithm.\n3. This paper is well-written and well-organized, and is easy to follow.", "reasons_to_reject": "1. This work appears to lean more towards an applied paper rather than an academic research paper. It extensively relies on existing techniques (e.g., Image caption model and ChatGPT), with limited original contributions.\n2. While the experimental results are promising, the credit for the success largely goes to the Large Language Model (LLM) employed. The paper, however, lacks an in-depth analysis from the perspective of MNER to explain why the performance is good, resulting in limited interpretability.\n3. Some recent relevant works have not been compared, such as:  1). M3S: Scene Graph Driven Multi-Granularity Multi-Task Learning for Multi-Modal NER;  2). Reducing the Bias of Visual Objects in Multimodal Named Entity Recognition.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces the FewMNER task for the first time and investigates the potential of the in-context learning paradigm in addressing this task. To adapt the in-context learning paradigm to the FewMNER task, this paper addresses three related problems and proposes a framework to accomplish this task. Through comparison with previous competitive methods, this paper's framework exhibits a significant advantage in this task. This paper also conducts extensive analysis experiments to reveal the impact of various factors on its performance and provides novel insights for future research.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bmczKC1NyF", "reviewer": "Reviewer_HAWy", "summary": "", "questions": "1. What are the practical applications of MNER, or in what scenarios can it be applied?\n2. Have you tried any other task demonstrations (Instruction construction in your paper) and executed a more rigorous comparison among these prompts?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you've taken to review our work.\n\nWe thank the reviewer for the valuable comments. Before we detail the response, we would like to clear our motivation and contribution. As described in section Introduction, we are the first to investigate MNER from a few-shot perspective and validate the feasibility of ICL for the FewMNER task. Besides, we propose a complete framework to solve the FewMNER task, analyze the impact of ICL on performance, and achieve satisfactory results. We sincerely hope that these clarifications provide a clearer understanding of the essence of our work. We kindly request that if you have any specific questions or suggestions pertaining to the contributions of our paper, please share them with us. Your expertise and insights are invaluable to our work.\n\n**Comment 1**\n\n>Speaking of few-shot settings for any task including MNER, people naturally come up with the scenarios where only a small number of training samples are available. So I cannot really understand why the author claims that they first propose the FewMNER task. I mean it could be a consensus among all researchers in AI field, or one could say that the author is simply the first one to put this concept into a paper, but the concept itself is known by anybody. And anyone will convert the images to natural languages if he/she wants to use LLMs in this task, so the point of utilizing a image caption model of others cannot be considered as an innovation either.\n\n**Response:**\n\nWe apologize for the confusion caused by our expression. To answer your confusion, here we explain why we claim that we first propose the FewMNER task. The reason is that the definition of our proposed FewMNER is different from previous few-shot tasks, which usually construct N-way K-shot setting to utilize meta-learning methods to solve few-shot tasks. Our proposed FewMNER task utilizes only K-shot. We want to distinguish it from previous work and emphasize our new setting. After our serious discussion, we think you are right and agree with you. Thank you for your thoughtful and valuable feedback. We decide to revise our expression in the next version.\n\n\nIn this paper, our main contribution is to validate the feasibility of ICL for the FewMNER task, propose a complete framework to solve the FewMNER task and achieve satisfactory results. The image caption is a sub-module of our framework, which we use to overcome the limitation of LLM that only accepts natural language as input.\n\n------\n\n**Comment 2**\n\n>Another thing is that the framework for FewMNER is too simple. You can find a large number of research or implementations which use LLM and ICL to accomplish NLP tasks, and the general system architecture is almost the same with the one illustrated in this paper. The solution is simply to prepare a task-specified prompt and choose the best ICL examples (in many cases, the most similar ones) as context.\n\n**Response:**\n\nThank you for your feedback. We agree with your viewpoint that there are many studies on the in-context learning (ICL) of LLM. However, most of them solve generation and classification problems, and solving extraction tasks in multimodal scenarios is still a challenge. Regarding our framework, we focus on multimodal and extraction characteristics to design effective modules (e.g., similarity rank, and output format). \n\n- For mitigating the bias of different modality models, we propose an efficient selection method based on text and image similarity ranks. Comparing sorting based on the sum of image and text similarity ranks and sorting based on the sum of image and text similarity scores, the former outperforms the F1 by 2.03% (54.08%->56.11%) and 2.67% (67.07%->69.74%) in 4-shot setting on $\\mathbf{\\mathcal{D}_{50}}$ set of two datasets, respectively (see w/ score in Table 3).\n\n- For adapting the extraction task, we design a dictionary-based format that explicitly defines the output structure as `{'PER':[], \"ORG\":[], \"LOC\":[], \"MISC\":[]}`. Compared with the sentence-based output format, the dictionary-based output format achieves 4.64% (51.47%->56.11%) and 8.17% (61.57%->69.74%) higher F1 in 4-shot setting on $\\mathbf{\\mathcal{D}_{50}}$ set of two datasets, respectively (see Table 10-11 in paper).\n\nWe hope that these effective and purposefully designed modules can differentiate our work from previous work. Additionally, we believe the simple and effective framework is helpful for future work.\n\n--------\n\n**Comment 3**\n\n>I had expected a brand new method of selecting ICL examples in multimodal scenarios, but the one provided is kind of disappointing, as it simply calculates the sum of cosine similarities between images and texts.\n\n**Response:**\n\nThank you for your constructive feedback. To demonstrate the effectiveness of our method, we compare it with a new method for selecting ICL examples. This method is from the paper: [Automatic Chain of Thought Prompting in Large Language Models (ICLR2023)](https://openreview.net/forum?id=5NTt8GFjUHkr). Following this paper, we first cluster text or image features into K clusters. Then, we sample the cluster center of each cluster, which aims to enhance the diversity of ICL examples. The experimental results are as follows. From the result, we can see that this clustering method performs lower than our method. This result demonstrates that our method is simple and effective.\n\n| Methods               | Twitter2015(F1%) | Twitter2017(F1%) |\n| --------------------- | :--------------: | :--------------: |\n| **Ours**              |      58.67       |      71.78       |\n| Cluster (Text)        |      57.83       |      71.35       |\n| Cluster (Image)       |      57.38       |      71.09       |\n| Cluster (Text, Image) |      57.19       |      71.10       |\n\n*Table 1: Different select ICL example methods in 4-shot setting on $\\mathbf{\\mathcal{D}_{50}}$ set. Note that the results presented here differ from paper due to OpenAI updating ChatGPT.*\n\n-----\n\n**[Q1]** What are the practical applications of MNER, or in what scenarios can it be applied?\n\n**[A1]** The MNER task can serve as the foundation for more advanced tasks, such as multimodal event extraction [1], multimodal aspect-based sentiment analysis [2], and building multimodal knowledge graphs [3]. For example, by using MNER, we can identify the entities that are related to a specific event or topic from social media data, and then analyze the sentiment, opinion, or attitude of the users towards those entities.\n\n[1]: [Joint Multimedia Event Extraction from Video and Article](https://aclanthology.org/2021.findings-emnlp.8) (Chen et al., EMNLP 2021)\n\n[2]: [AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis](https://aclanthology.org/2023.findings-acl.519) (Zhou et al.,ACL 2023)\n\n[3]: [MMEKG: Multi-modal Event Knowledge Graph towards Universal Representation across Modalities](https://aclanthology.org/2022.acl-demo.23) (Ma et al., ACL 2022)\n\n**[Q2]** Have you tried any other task demonstrations (Instruction construction in your paper) and executed a more rigorous comparison among these prompts?\n\n**[A2]** Yes, when designing the framework, we tested the performance of different instructions and chose the one that performed better. To answer your concerns, we design four types of instruction and compare their performance. The details are as follows. From the experimental results, we can see that more detailed instruction descriptions lead to better performance.\n\n- Instruction (paper): You are a smart and intelligent Multimodal Named Entity Recognition (MNER) system. I will provide you the definition of the entities you need to extract, the sentence from where you extract the entities, the image description from image associated with sentence and the output format with examples.\n\n- Instruction (v2): Please extract the entities from text.\n\n- Instruction (v3): Do multimodal named entity recognition task.\n\n| Instruction     | Twitter2015(F1%) | Twitter2017(F1%) |\n| --------------- | :--------------: | :--------------: |\n| Instruction (paper)         |      58.67       |      71.78       |\n| Instruction (v2) |      57.34       |      71.91       |\n| Instruction (v3) |      57.72       |      70.27       |\n| No Instruction  |      55.60       |      71.03       |\n\n*Table 2: Different instruction in 4-shot setting on $\\mathbf{\\mathcal{D}_{50}}$ set. Note that the results presented here differ from paper due to OpenAI updating ChatGPT.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-written and very easy to follow, with few grammar mistakes or typos.", "reasons_to_reject": "This paper appears to be poorly lacking in innovation. It seems to me that in the era of LLM, anyone who wants to utilize LLMs to accomplish MNER task is very likely be able to carry out the research described in this paper in a method quite similar to the method outlined here. The reason is as follows:\n1. Speaking of few-shot settings for any task including MNER, people naturally come up with the scenarios where only a small number of training samples are available. So I cannot really understand why the author claims that they first propose the FewMNER task. I mean it could be a consensus among all researchers in AI field, or one could say that the author is simply the first one to put this concept into a paper, but the concept itself is known by anybody. And anyone will convert the images to natural languages if he/she wants to use LLMs in this task, so the point of utilizing a image caption model of others cannot be considered as an innovation either.\n2. Another thing is that the framework for FewMNER is too simple. You can find a large number of researches or implementations which use LLM and ICL to accomplish NLP tasks, and the general system architecture is almost the same with the one illustrated in this paper. The solution is simply to prepare a task-specified prompt and choose the best ICL examples  (in many cases, the most similar ones) as context. \n3. I had expected a brand new method of selecting ICL examples in multimodal scenarios, but the one provided is kind of disappointing, as it simply calculates the sum of cosine similarities between images and texts. ", "questions_to_authors": "1. What are the practical applications of MNER, or in what scenarios can it be applied?\n2. Have you tried any other task demonstrations (Instruction construction in your paper) and executed a more rigorous comparison among these prompts?\n", "paper_topic_and_main_contributions": "This paper focuses on the multimodal named entity recognition (MNER) task, which aims to detect named entities in a given text with the assistance of additional images. Furthermore, it concentrates on the field of few-shot multimodal named entity recognition (FewMNER) task, which is set in a scenario where only a few labeled text-image pairs are available for training. \nThe contributions of this paper are as follows:\n1. This paper is the first paper to propose the FewMNER task and explore the potential of the in-context learning paradigm for this task.\n2. To adapt the in-context learning paradigm to the FewMNER task, the paper addresses three related problems including the LLMs do not receive visual inputs, selecting useful examples for ICL in multimodal scenarios has not been approached and the absence of good demonstration designing. And they proposed a framework to accomplish this task, which includes: \n- a retrieve module, which calculates the sum of text cosine similarity and image cosine similarity between test examples and candidate examples. The paper selects the top-k examples of the similarity ranking (k-nearest) as the examples to be used in ICL. \n- a demonstration designing module including instruction construction, which helps the LLM understand the task and the entity labels, and demonstration construction, which contains image description, sentence , and output (ICL examples).\n- a predict module which is simply feed the concatenation of instruction, demonstration and the test sample into LLM and get predicted output sequence.\n3. Through comparison with previous competitive methods, the framework in this paper exhibits a significant advantage in this task. The authors conduct extensive analysis experiments to reveal the impact of various factors on its performance and provide novel insights for future research.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper builds a few-shot learning based model for multimodal NER.  There has been a very little attempts in this area, and hence it makes a contributions. There are certain aspects that need to be addressed: few shot setting for the NER; use of image captions and its impact on the overall model performance, and the use of ChatGPT in proper way.
Decision: Accept-Findings