ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition
Authors: Xiujiao Li
Guanglu Sun
Xinyu Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal named entity recognition
Local visual information
Global visual information
TL;DR: The paper introduces a novel module named Entity Spans Position Visual Regions (ESPVR) to identify the most relevant visual regions corresponding to entities in text, which improves the efficiency of multimodal named entity recognition.

Abstract: Multimodal Named Entity Recognition (MNER) uses visual information to improve the performance of text-only Named Entity Recognition (NER). However, existing methods for acquiring local visual information suffer from certain limitations: (1) using an attention-based method to extract visual regions related to the text from visual regions obtained through convolutional architectures (e.g., ResNet), attention is distracted by the entire image, rather than being fully focused on the visual regions most relevant to the text; (2) using an object detection-based (e.g., Mask R-CNN) method to detect visual object regions related to the text, object detection has a limited range of recognition categories. Moreover, the visual regions obtained by object detection may not correspond to the entities in the text. In summary, the goal of these methods is not to extract the most relevant visual regions for the entities in the text. The visual regions obtained by these methods may be redundant or insufficient for the entities in the text. In this paper, we propose an Entity Spans Position Visual Regions (ESPVR) module to obtain the most relevant visual regions corresponding to the entities in the text. Experiments show that our proposed approach can achieve the SOTA on Twitter-2017 and competitive results on Twitter-2015.

ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition
Xiujiao Li Guanglu Sun Xinyu Liu
Harbin University of Science and Technology. Harbin, China
2120410084 htw.:uuuuttuurrbbu.c.u..... *corresponding author: sunguanglu @ hrbust.cdu.cn
2010400001 @ stu.hrruut.edu.cn
Abstract
aims to identify and classify named entities from text using images as auxiliary information. MNER can disambiguate multi-sense words by augment- ing linguistic representations with visual informa- tion, resulting in superior performance compared to traditional Named Entity Recognition (NER).
Multimodal Named Entity Recognition (MNER) uses visual information to improve the performance of text-only Named Entity Recognition (NER). However, existing methods for acquiring local visual information suffer from certain limitations:
(1) using an attention-ban based method to extract visual regions related the text from visual regions obtained through convolutional architectures (e-g., ResNet), attention is distracted by entire image, rather than being fully focused on the visual regions most relevant to the (e.g..
While previous efforts have yielded promising results, they still fall short in effectively selecting visual information. For existing methods of uti- lizing visual information, we classify it into two types: global visual information and local visual information.
text: (2) using an object detection- Mask R- R-CNN) method detect visual object regions related to the text, object detection has a limited range of recognition categories. Moreover, the visual regions obtained by object detection may not correspond to the entities in the text. In summary, the goal of these methods is not to extract the most relevant visual regions for the entities in the text. The visual regions obtained by these methods may be redundant or insufficient for the entities in the text. In this paper, we propose an Entity Spans Position Visual Regions (ESPVR) module to obtain the most relevant visual regions corresponding to the entities the text. Experiments show that our proposed approach can achieve the SOTA on Twitter-2017 and competitive results on Twitter-2015.
Some previous works (Lu et al., 2018; Zhang et al., 2018; Yu et al., 2020; Chen et al., 2021; Sun et al., 2021, 2020; Liu et al., 2022a,b; Wang et al., 2022) consider that if the whole image information input the multimodal interaction module, then such image information is global visual informa- tion. However, the multimodal interaction module relies on attention to select the visual regions as- sociated with the text for interaction. Therefore, the visual information that eventually interacts with the text is mainly the local visual information lated to the text. In other words, even if the input the multimodal interaction module is the whole image information, finally the local visual informa- tion is selected using an attention-based method and then interacts with the text, so we consider it as a process of using attention to select the local visual information. When using attention to extract visual regions, attention is distracted by the entire image, rather than being fully focused on the vi- sual regions most relevant the text. Therefore, using an attention- method select the local visual information only obtains valuable visual information but also introduces irrelevant visual information.
1 Introduction
Named entity recognition (NER) a fundamental task in the field of information extraction, which can automatically recognize named entities in text and classify them into predefined categories. NER has been widely used for many downstream tasks, such entity linking and relationship extraction. With the rapid development of social media, mul- timodal deep learning is widely used to perform structured extraction from massive multimedia news and web product information. Among them, Multimodal Named Entity Recognition (MNER)
Besides, most of the previous approaches (Wu et al., 2020; Wang et al., 2020; Zheng et al., 2020; Zhang et al., 2021: Wang et al., 2021) use object detection (e.g., Mask R-CNN) to detect visual ob-

---

ject regions, and treat visual objects as local visual information to interact with the text. However, ob- ject detection has a limited range of recognition categories, so it may not detect all objects within the categories defined by the dataset. Moreover, the visual regions obtained by object detection may not correspond to the entities in the text.
tecture (e.g.. ResNet), and then using the multi- modal interaction module with attention to select the visual regions associated with the text.
In fact, not all visual regions within an image are beneficial to improve the accuracy of the model pre- diction. To address this problem, some researchers proposed a method of first dividing the whole im- age into multiple visual regions equally, and then extracting the most relevant visual regions to the text in order to filter out irrelevant visual informa- tion. The visual information that ends up interact- ing with the text is actually the local visual infor- mation after filtering, even if the input is global visual information. Lu et al. (2018) used the pre- training ResNet model to extract visual regions and then added them to the text embedding by a vi- sual attention model. To make full use of text and visual information, Zhang et al. (2018) used adap- tive co-attentive networks to fuse text embedding and visual regions representation. Yu et al. (2020) proposed a multimodal interaction module to ob- tain both image-aware word representations and word-aware visual representations, and used text- only entity spans detection as an auxiliary module to mitigate visual bias. Chen et al. (2021) used an external knowledge database to obtain the fi- nal multimodal representation by attention-guided visual layer.
In summary, the goal of these methods is not to extract the most relevant visual regions for the entities in the text. The visual regions obtained by the above two methods may be redundant or insuf- ficient for entities contained in the text, leading to identifying non-entity as an entity or incorrectly predicting an entity category. Therefore, to obtain the most relevant visual regions for the entities in the text, we propose an Entity Spans Position Vi- sual Regions (ESPVR) module. Specifically, the ESPVR module consists of two modules: Entity Spans Identifying (ESI) module, Visual Regions Positioning (VRP) module. First, the ESI module identifies all entity spans in the text. Then the VRP module uses these entity spans to extract entity features and uses the entity features to locate the visual regions that are most relevant to the entities in the text.
To summarize, the major contributions of our paper are as follows:
We propose novel ESPVR module for MNER, which can select the most relevant visual regions for the entities in the text. The ESPVR mod- ule consists of two modules: Entity Spans Identify- ing (ESI) module, and Visual Regions Positioning (VRP) module.
Irrelevant text-image pairs account for large proportion of the dataset. Therefore, Sun et al. (2021) and Sun al. (2020) used modified BERT encoder obtain information for inter-modal fu- sion and then introduced text-image relationship classification as a subtask to determine whether image features were useful. In addition, Liu et al. (2022a) proposed a novel uncertainty-awarr frame- work for social media MNER
We conduct extensive experiments on two benchmark datasets, Twitter-2015 and Twitter- 2017, to evaluate the performance of our ESPVR module. Experimental results show that the ES- PVR module outperforms the current ofate-of-the- art models on Twi1 and yields competitive results on T015
In addition, for the problem of fine-grained se- mantic correspondence between objects in images and words in the text. Liu et al. (2022b) performed an enhanced representation of each word in the text by semantic enhancement and performed cross- modal semantic interaction between text and vision at different visual granularities. Wang et al. (2022) proposed a Scene graph driven Multimodal Multi- granularity Multitask learning framework.
2 Related work
general, studies about MNER are similar in terms of text feature extraction. However, there are differences in research methods when using image information and fusing modal information. The existing work can be classified into the fol- lowing two categories based the use of visual information:
(2) Obtaining visual objects from the whole im- age by object detection (e.g.. Mask R-CNN), and treating visual objects as local visual information to interact with the text.
(1) The entire image is equally segmented into multiple visual regions by the convolutional archi-
Visual objects are considered fine- line-grained image

---

B- PER
B- ORG
B-
ORG
CRF Decoding
hoot
Multinodal Interaction Module
ESI moddlle
VRP musdale
bi
bi
Span CRF Decoding
b.
bd
Tari
Selecting
entity features
Transformeer Layer with Self-Atteation
Calculating
ESPVR
modhde
attention
bo
b144
BERT
Swin Transformer
Pearce powers
a couple of
Rays finale
baseball players
[CLS]
[SEP]
standing
[SEP]
Dodgers #
each other
baseball
Text
Image Caption
Visual
Figure 1: Overall Architecture of Our ESPVR.
representations, and for the text with multiple en- tity types, the visual object regions of the related images can be used to capture different entity in- formation. Wu et al. (2020) used Mask RCNN for object detection, they embedded top-k objects into vectors to interact with text features through the dense co-att co-attentive module. In addition to this, Wang et al. (2020) proposed multimodal align- ment framework that used contrastive objective, guide alignment between visual and textual rep- resentations. Zheng et al. (2020) used an adver- sarial learning technique that aimed to fuse textual and visual features into a common fcature space. For the same problem, Zhang et al. (2021) first ex- tracted noun phrases in the text and visual object regions in images and then used GNN to model the relations between them.
prove performance. Based on this work, Wang et al. (2021) aligned image features into the text space by using image-text alignment to better uti- lize the attention mechanism transformer-based pre-trained textual embeddings.
Method
In this section, we first introduce the task definition. and then describe the proposed model in detail.
Given text and image pair (X; V) as input, where X {a), ..., ; and n denotes the length the text, the goal MNER is to extract a set entities from and classify each entity into one of the pre-defined categories with the assistance of image information, e.g., Person (PER), Location (LOC),, Organization (ORG), and Miscellancous (MISC). As with most existing work in MNER, regard the task sequence labeling problem. Specifically, let {V1 V/. 3n3 rep- resent a label sequence corresponding to X, where 3/4 E Ç and Ç is the pre-defined label set with stan- dard BIO2 tagging schema.
Task Formulation:
Although many multimodal neural techniques have been proposed to incorporate images into the MNER task, the ability of models to exploit mul- timodal interactions remains unclear. Therefore, Chen al. (2020) provided in-depth analysis existing multimodal fusion techniques from dif- ferent perspectives and described a situation where the use of image information did not always im-

---

3.1 Overall Architecture
are the 12 12 144 visual region features di- vided by Swin Transformer, and each region is represented by 1536-dimensional vector.
Fig. 1 illustrates the overall architecture of the ES- PVR, which consists of four major modules: (1) Feature Extraction module; (2) Entity Spans Po- sition Visual Regions (ESPVR) module; (3) Mul- timodal Interaction module; (4) CRF Decoding module.
3.3 Entity Spans Position Visual Regions (ESPVR) Module
As shown on the left side of the ESPVR module in Fig. 1, we first use a transformer layer with self-attention (Vaswani et al., 2017) to capture the intra-modality relation for the text modality and obtain each word's s textual hidden representation T {Lo, (2+1) where 4 E R denotes the generated hidden representation for Ię-- Entity Spans Identifying (ESI) module: The purpose of the EBI is to identify the position of the head and tail of the entities in the text, which can be used for positioning the visual regions that are most relevant to the entities in the text.
We first obtain word representations and visual representations, respectively. Then, to obtain local visual information, we deploy Entty Spans Posi- tion Visual Regions (ESPVR) module to position the visual regions that are most relevant to the en- tities in the text. Next, a Multimodal Interaction module is devised to fully capture cross-modee mods--md semantic interaction between textual hidden repre- sentation and visual regions hidden representation. Finally, the CRF Decoding module assigns tity label to each word in the input sequence, lever- aging the hidden representations obtained from the Multimodal Interaction module.
We remove the type information and define the of span labels Z' [B,I,0, and use Z {21, .n.2 to denote the sequence of labels, where 2j C Z'. Subsequently, T fed into the Span_ CRF decoding layer to predict a sequence Z of labels of X.
3.2 Feature Extraction Module
Word Representations: To better model the semantic information of text X and get different representations for the same word in different contexts, we leverage pre-trained language model BERT (Devlin et al., 2018) as our text encoder. Moreover, image captions from an image caption- ing model can fully describe the whole image and provide more semantic information. Therefore, to let the text learn contextual information from an image caption, we first use [SEP] concatenate the text and image caption as cross-modal input view rather than the text-only input view. Then, we denote the input as X/ = {[CLS], x1, T..,,n, [SEP], Ih, D/, [SEP]] where i is the word of the text, 2nd is the word of the image caption, [CLS] and [SEP] are special tokens of BERT. Lastly, the text X is fed into the BERT to get the word contextualized representa- tions C {Co, C2, Cn 1} is the generated contextualized representation for I; e; E IRd, d stands for the dimension of the word embedding. Visual Representations: To obtain better vi- sual representations, we use the pre-trained model Swin Transformer (Liu et al., 2021). Specifi- cally, given an image V. the visual representations B (bg, b1, ..., )r are obtained by extracting the output of the last layer of Swin Transformer, where by represents the feature of the whole image, usually used for image classification, (b.....,b,.. b144)
Visual Regions Positioning (VRP) Module: Af- ter obtaining the sequence Z of labels of X, we first need to select entity features E {er, e2, .... em) corresponding to entities from T based on the la- bels in Z, where 6 stands for the feature of the i i- th entity, m stands for the number of entities in- cluded in the text. Then, to maintain the same scale as most of the original images, we extract visual region feature of size x a that is most relevant to the entities in the text. The specific process is as follows:
First, we select a visual region feature be from (b1, (.., b14 that most relevant to the entities the text. Specifically, we take each entity feature in E = {ei, E2, ..., em as Q, each visual region feature in relation score F(c,,b,) between each visual region feature and each entity feature, and sum up the cor- relation scores of each visual region feature and all entity features to obtain 1 144 correlation scores. We select a visual region feature be correspond- ing to the maximum value from the sum of 144 correlation scores:
b144} as K, and calculate the cor-
QKT
F(es,b,) = softmax(
(1)
Vdk
b. = =arg max (E F(e),)))
(2)
15j$144

---

where d3 is the dimension of key vector K.
where CA₁, is the i th head of Cross-Modal At- tention, { Wki, W Rd/mxd refers the weight matrices for the Q, K, and respectively, and W E Idddd multi-head attention. Then, we ob- tain the output P = (Po.PI. P.o.PP........ Pn 11) of the first CMT layer:
Then, we select neighboring visual region fea- tures based on the index с of be. Specifically, the positions of all visual region features are expressed in row and column coordinates, and visual region features with row and column distances less than a are identified neighboring visual region features of visual region feature be, where the range of is 1 S < 11. Because the number of all visual regions is 12 x 12, the upper limit of a is 11. Next, we select neighboring visual region fea- ture by that is most relevant to be from neigh- boring visual region features. Specifically, we take the visual region feature be as Q and each neigh- boring visual region feature as K, use attention to calculate the correlation score between be and each neighboring visual region feature, and select neighboring visual region feature b with the largest correlation score.
P LN( + MA(,,
(5)
P= LN(P +
(6)
where LN and FFN stand layer normalization and feed-forward network respectively. In the sec- ond stage, we treat T as Q, P as K and V. Then the second CMT layer generates the image-aware word representations A = (ao, .1....; On 1 11.
To obtain word-ar wwar- visual representations, we use CMT layer to perform basic-level semantic interaction. We treat T Q. and as K and V. Then the word-aware visual representations Q (40- 9n+1) can be computed through Equation. 3-Equation. 6.
Finally, to maintain the same scale as most of the original images, we extract a visual region fea- ture of size a x a. Specifically, we compare the coordinate of the visual region feature be and the coordinate of the neighboring visual region fea- ture bt, and select minimum number of rows and columns the coordinates of the top-left visual re- gion feature by for the visual region feature And using as the edge length of c, we add a to the rows and columns of coordinate é to get the coor- dinate d of the down-right visual region feature ba for visual region feature v. We use all the visual region features that lie within the range of and d to form an overall visual region feature v.
To trade off the cross-modality contributions, we use a gate function to obtain the final semantic interaction representation H = {ho, ha+1)
g = = WTQ)
(7)
H Comcat(A,g) Q)
(8)
where A image-aware word representa-
tions, Q are word-aware visual representations. (Wa. Wr] E IRdxd refer weight matrices, and a stands the element-wise sigmoid function.
3.5 CRF Decoding Module
3.4 Multimodal Interaction Module
Conditional Random Fields (CRF) take into ac- count the correlations between labels in neighbor- ing positions and assign score to the entire quence of labels. This approach can lead to im- proved accuracy in sequence labeling tasks. Conse- quently, given a sequence X, all the possible label sequences can be produced as follows:
Following (Yu et al., 2020), we stack the cross- modality Transformer layers to learn the cross- modal interaction between the words and visual regions. The components of the cross-modality Transformer (CMT) layer are the same as the Trans- former.
To obtain image-aware word representations, we stack two CMT layers to perform superior-level se- mantic interaction. These two CMTs are internally calculated in the same way, except that the Q. K, and V are from different sources. In the first stage, we first perform multi-head Cross-Model Attention by treating v as Q, and T as K and V:
II S,
P(y EyEY I1 S. 1.99,
(9)
where S, Wi, X) and S, X) are po- tential functions.
3.6 Model Training
CA(e, T) soft max( [W..T] [W..T'" / d/mm
There are two tasks our proposed ESPVR model:
(3) MNER, and ESI. In the training phase, we jointly
= = (an.o.........All.... Cy,
(4) train the whole model. The final training objective

---

function L is the combination of MNER loss and ESI loss. By minimizing negative log-likelihood estimation, L can be denoted as:
Nvidia GeForce-RTX 2080 GPU with batch size 32.
4.2 Baseline methods
L = LMNER + LES! = - (log(P(y)X)) - A Aog(((s((X))
We compare our ESPVR with several base- line models for comprehensive comparison to demonstrate the superiority of our ESPVR. For unimodal, we choose BiLSSM-CRF (Huang et al., 2015), BLSTM-CNNS-CRT (Ma and Hovy, 2016), HBiLSTM-CRF (Lample et al., 2016), BERT+softmax (Devlin et al., 2018), BERT+CRF (Dai
(10)
where À is a hyperparameter to control the contri- bution of the auxiliary ESI module. Here we set to 0.08
4 Experiments
In the following section, we conduct experiments on two MNER datasets, comparing our Entity Vi- sual Regions Positioning (ESPVR) approach with several unimodal and multimodal approaches.
(Devlin et al., 2018), BERT-BiLSTM et al., 2019), For multimodal, we choose GVATT- HBiLSTM-CRF (Lu et al., 2018), AdaCAN-CNN- BiLSTM-CRE (Zhang al., 2018). GVATT-BERT- CRF (Lu al., 2018), UMT-BERT-CRF (Yu al., 2020), MAF (Wang al., 2020), UMGF (Zhang et al., 2021), ITA (Wang et al., 2021), UAMNer (Liu et al., 2022a), MGCMT (Liu et al.. 2022b).
4.1 Experiment Settings
Datasets: We use two publicly MNER datasets (Twitter-2015 (Zhang et al., 2018) and Twitter-2017 (Lu et al., 2018) to evaluate the effectiveness of our framework. Twitter-2015 and Twitter-2017 in- clude multimodal tweets from 2014 to 2015 and from 2016 2017 respectively. Both datasets are composed of four types of entities: Person (PER), Location (LOC), Organization (ORG), and Mis- cellaneous (MISC) (In Twitter-2017, the last tag is Other. Here, we collectively refer to them as MISC). Each sample in the two datasets is com- posed of a pair sentence, image.
4.3 Main Results
Following the other baselines, we employ standard precision (P), recall (R), and F1 score (F1) to evalu- ate the overall performance and report FI for every single type of metric. Since the two Twitter datasets differ significantly in type distribution and data characteristics, we also conduct extensive experi- ments in the self-domain and cross-domain cases to demonstrate the validity and generality of our approach.
Implementation Details: For both datasets, the same hyperparameters. To compare each unimodal and multimodal method in the experi- ment, the maximum length of the text set to 128 which can cover all words. For our ESPVR ap- proach, most of the hyperparameters are set in the following aspects: The word representations C are initialized with the pre-trained BERT (bert-base- uncase) model of dimension 768 by Devlin et al. (2018), and fine-tuned during training. The visual embeddings B are initialized by Swin Transformer with the dimension of 1536. Swin Transformer is fixed during training. The Self-Attentio layer has a head size of and a number of 4. Additionally, the Cross-Modal Attention has feature dimensions of 512. The learning rate, the dropout rate, and the tradeoff parameter are respectively set to le-4, 0.4, and 0.08, which achieve the best perfor- mance on the development set of both datasets via small grid search over the combinations of [le-5, le-4], 10.1, 0.5], and [0.05, 0.9]. We implement the proposed model with PyTorch (Paszke et al., 2019). The model trained and tested on one
Self-domain Scenario. Table 1 shows the over- all results of unimodal and multimodal approaches on the two benchmark Twitter MNER datasets. From the table, have the following findings:
(1) Pre-trained model BERT is more powerful than conventional neural networks. This indicates that the pre-trained model can indeed provide abun- dant syntactic and semantic features. CRF con- siders the correlations between labels in neigh- borhoods and scores the whole sequence of la- bels. Therefore, the recent approaches are typically based on BERT-CRF.
(2) Multimodal approaches can usually perform better than their corresponding unimodal baselines. By comparing all multimodal and unimodal ap- proaches, we can find that both global images and visual information of local objects are valuable MNER This confirms that visual information can bring wealth of external knowledge to the text. However, this approach does not bring very signifi- cant improvement, which demonstrates that MNER

---

Twit
Twitter20115
Single
TypetF])
Overall
Single Type(H1)
Overall
Modality
Methods
PER
LOC
ORG
MISC
P
R
F1
PER
LOC
ORG
MISC
F1
BI-LSTM-CRF
85.12
72. 68
72.30
52.56
79.42
73.43
76.31
76.77
72.56
41.33
26
68. 08.14
61.09
64,44
BLSTM CNNS CRF
87,99
77.44
74.02
78.76
79.37
80.86
47.77
32.61
66.24
68.09
HRILSTM-CRF
87.91
78.57
76.67
59.32
82.69
78.16
80.37
82.34
76.83
51.59
32.52
70.32
68.05
69.17
Text
BERT+sofimas
90.88
84,00
79.25
61.63
82,19
83.72
82.95
84.72
79.91
58.26
38.81
68.30
74.61
71.32
RERT&CRF
90.06
81.99
81.83
63.41
82,98
84,46
84.74
80.51
60:27
37.29
69.22
74.59
71,81
BERT BILSTM CRF
90.29
84.55
80.97
64.85
83,20
84.68
83.93
84.32
79.31
61,66
37.53
71.03
73,57
72.27
GVATT-HBiL STM-CRF
89.34
78.53
79.12
62.21
80.38
81.87
82.66
27.21
55.06
35.25
73.96
67.90
70,80
Ad:CAN- CNN BILSTM-CRE
89.63
77.46
79.24
62.77
84.16
80.24
82.15
81.98
78.95
53.07
34,02
72.75
68.74
70.69
GVATT-BERT-CRE
90.94
83.52
81.91
62.75
83.64
84.38
84.01
84.43
80.87
59.02
38,
69:15
74.46
71.70
BLLCAN-BERT-CUE
90.20
82.97
82.67
38.98
69.34
75.22
72,16
UMT-BFRT-CRF
91.56
84.73
82.24
70.10
85,28
85.34
85,31
85.24
81.58
63.03
39.45
71.67
75.23
73.41
Image
UMGF
91.92
85.22
83.13
69.83
86.54
84,50
85,51
84.26
83.17
62.45
42.42
74.49
75.21
74,85
ITA
91.40
84.80
84.80
84.00
68.60
85.72
85.60
82.60
64.40
44.80
75.60
UAMNNr
91.86
85.71
84.25
68.73
86.17
86.23
86,20
65.14
81.66
62.46
40.95
73.02
74.75
73.87
91.51
85.80
85.10
68.79
86.13
86.38
86.25
84.67
81.18
63.35
41. 11.82
71.86
75,10
73,42
MOCMT
90.82
86.21
86.26
66.88
86.03
86.16
86,09
85.84
82.03
63.08
40.81
73.57
75.59
74.57
M38
92.73
84.81
82.49
69.53
86,93
85.21
86.06
86.05
32
62.97
41. 36
74.92
75.14
75.03
Ours
ESPVR
92.73
84.75
83.82
70.55
85.70
86.52
39.45
38.08
71.94
74.17
Table 1: Results on Twitter-2017 and Twitter-0015.
still has considerable space for progress in propos- ing a more effective multimodal approach.
4.4 Ablation Study
To show the effectiveness of each component in ESPVR, we conduct an ablation study by removing the particular component from it. Table 3 shows comparison results between the full model and its ablation methods.
(3) Our ESPVR approach achieves tate-offthe- art performance Twitter-201 2017 dataset and com- petitive results
Twitter-2015. To position the visual regions that are most relevant to the entities in the text, we design an ESPVR module. In com- parison with the existing multimodal methods, our approach outperforms the state-of-the-an MAF by 0.27 on Twitter-2017 but performs slightly worse on Twitter-2015. This is because there are many unmatched text-image pairs, and it is one direction of our future work.
Telee/1177
Slylk Doa
Ouak
Sugi TpeFi
Chopal
9:01
$4.84
6839
H
900
21.8
1140
13.99
10:22
6035
85
643
605
1039
1232
1345
12.9
Table 3: Ablation Study of ESPVR.
w/o IC. This approach completely ignores the global information brought by image captions. We remove the image captions, resulting in reduced performance, which shows image captions can add external support for each input word.
Cross-domain Scenario. Table 2 shows a per- formance comparison of our ESPVR approach with UMT and UMGF in a cross-domain sce- nario. Twitter-2011 2017 - Twitter-2015 denotes that the trained model on Twitter-2017 is further used to test Twitter-2015. Similarly, Twitter-2015 + Twitter=2017 denotes that the trained model on Twitter-2015 further used to test on 2011 From this table, we can observe that our ESPVR approach significantly outperforms UMGF by 0.38 and 0.84 on the overall F1, respectively. These re- sults further confirm the effectiveness of our model.
w/o ESPVR. This approach completely ignores the problem of fine-grained semantic correspon- dence between the semantic units in the text-image pair. When we remove the ESPVR module and solely train the main MNER task, there is a notice- able decline in overall recall and Fl scores, along- side a slight improvement in overall precision. The result is consistent with our hypothesis that visual regions can provide clues for fine-grained semantic interaction.
Pig
1337
11
1857
cen
K
w/o IC + ESPVR. This approach completely ignores global visual information and local visual information. We remove image captions and the ESPVR module, resulting in significant degrada- tion of the performance of the model, indicating that both image captions and the ESPVR module
1N27
48.44
12.38
4700
3317
43.59
54.5
63.55
31,77
35.3 M087 1518
1500
61
6948
3121
12:35
1130
5040
Table 2: Performance comparison in the cross-task scc-
nario.

---

ROADTRIP PLUUS
POPS 2013
PMM 9PM MAY 5-8- INSMETRE
IS
A. Now that we are in 2017, I' I'm looking forward to warmer weather and @ [elli uelliuuttebcckkk PERJI's soccer matches!
B. Don't miss out on this year's AMAZING Pops concert @ CDS! Buy your tickets tomorrow before they sell out!
C. Thanks @ |newbalance ORGjI for these colorful shirts for our recreational soccer program. we love them!
UMT:
1-PER V
CDS ORG
1-PER x
UMGF:
1-PER V
CDS PER
1-PER
AMAZING Pops MISC x
MGCMT:
I-PER
1-ORG
CDS PER x
our:
1-PER V
1-ORG v
Table 4: The second row shows a few representative samples from the test set and their manually labeled entities. The bottom four rows show the prediction results of different methods for these test samples.
are essential in our framework. Removing image captions has slightly greater impact than remov- ing the ESPVR module. This is probably because some images are not relevant the text. Over- all, the different components of our model work effectively with each other to produce a better per- formance of the model in the MNER task.
this text does not contain entity, and it should not provide local visual information for this text under normal situations. However, the existing methods for obtaining local visual information all obtain large amount of visual information from this image. So, those methods identify a non-entity as an entity. Another example is Table 4.C, UMT and UMGF use the error guidance of local visual information about the Person from the image and omit the relationship between Person and Person, resulting in the identification of "newbalance" "PER". On the contrary, MGCMT and our method can accurately determine the entity. Here A can get the correct result because uses the local visual information obtained both ways, thus refining the local visual information.
4.5 Further Analysis
To validate the effectiveness and robustness of our method, we conduct further analysis with three specific examples Table 4.
For informal or incomplete text, if the corre- sponding visual information is provided, the visual context will provide useful clues to the text. For example, in Table 4.A, the image's most obvious local visual information is the person, and all meth- ods can obtain this local visual information with significant features. Therefore, all the multimodal approaches can correctly classify their types after incorporating the image.
Conclusion
In this paper, we present an Entity Spans Position Visual Regions (ESPVR), which obtains the most relevant visual regions for the entities in the text as fine-grained local visual information. The ex- perimental results reveal the superiority of fine- grained local visual information acquired through this method. This information proves more advanta- geous in enhancing the performance Multimodal Named Entity Recognition (MNER) compared to
It is essential to obtain local visual information from the image that is relevant the entities in the text. If the obtained local visual information is redundant or insufficient, may result in misidenti- fying a non-entity as an entity or incorrectly predict- ing the entity category. For example, in Table 4.B, E,

---

using attention-based and object detection-based methods.
Guillaume Lample. Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint 1633v01603.01360
Limitations
Although our experiments demonstrate the effec- tiveness of our method, there are still some limi- tations that can be improved in future work, First, data augmentation is a necessity to enhance data efficiency in deep learning. Our model lacks multi- modal data enhancement. Second, there are many irrelevant text image pairs in the data of MNER, and our method aims to solve the problem of acquir- ing local visual information. By filtering out text- irrelevant images prior to obtaining local visual information and focusing solely on text-relevant images for acquiring local visual regions, the effi- cacy of our proposed method is likely to be further amplified. We hope that the insights from this work will stimulate further research on MNER perfor- mance.
Luping Liu, Meiling Wang. Mozhi Zhang, Linbo Qing. and Xiaohai He. 2022a. Uamner: uncertainty-awaal multimodal named entity recognition in social media posts. Applied Intelligence. 52(4):4109 4125.
Peipei Liu, Gaosheng Wang, Hong Li, Jie Liu, Yimo Ren, Hongsong Zhu, and Limin Sun. 2022b. Multi- granularity cross-modality representation learning for named entity recognition on social media. arXiv preprint arXiv:2210. /4163.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF international conference on computer vi- sion. pages
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang. and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990-1999
Acknowledgements
This study is in part supported by the Key Research and Development Project of Heilongjiang Province (2022ZX01A) A33),, the 2020 Heilongjiang Province Higher Education Teaching Reform Project (SJGY 20200320).
Xuezhe Ma and Eduard Hovy, 2016. End-to-ent sequence labeling via bi-directional Istm-cnns-crf. arXiv preprint arXiv: 1603.01354.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
References
Dawei Chen, Zhixu Li, Binbin Gu, and Zhigang Chen. 2021. Multimodal named entity recognition with im- age attributes and image knowledge. In Database Systems for Advanced Applications: 26th Interna- tional Conference, DASFAA 2021, Taipei, Taiwan, April 11-14, 2021, Proceedings, Part II 26, pages 186-201. Springer.
Lin Sun, Jiquan Wang, Yindu Su, Fangsheng Weng. Yuxuan Sun, Zengwei Zheng, and Yuanyi Chen. 2020. Riva: pre-trained tweet multimodal model based on text-image relation for multimodal ner. In Pro- ceedings of the 28th International Conference on Computational Linguisties, pages
Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. 2020. Can images help recognize entities? a study of the role of images for multimodal ner. Xrxiv preprint ar arXiv:2010. 12712.
Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fang- sheng Weng. 2021. Rphert: text-image relation propagation-base bert model for multimodal ner. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13860-13863
Zhenjin Dai, Xutao Wang, Pin Ni, Yuming Li, Gangmin Li, and Xuming Bai. 2019. Named entity recogni- tion using bert bilstm erf for chinese electronic health records. 2019 12th international congress on im- age and signal processing, biomedical engineering and informatics ceisp-bmei), pages 1-5, IEEE
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint arXiv:1810.04800
Jie Wang, Yan Yang. Keyu Liu, Zhiping Zhu, and Xi- aorong Liu. 2022. M3s: Scene graph driven multi- granularity multi-task learning for multi-modal ner. IEFE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 31:111-120.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015, Bidirec- tional Isum-crf models for sequence tagging. arXiv preprint arXiv::558.0099:

---

Qinxin Wang, Hao Tan, Sheng Shen, Michael Ma- honey, and Zhewei Yao. 2020, Maf: Multimodal alignment framework for weakly-supervised phrase grounding. arXiv preprint arxiv:2010.05379.
Xinyu Wang. Min Gui, Yong Jiang, Zixia Jía, Nguyen Bach, Tao Wang, Zhongqiang Huang. Fei Huang, and Kewei Tu. 2021. Ita: Image-text alignments for multi-mode named entity recognition. arXiv preprint urXiv:2112.06433
Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. 2020. Multimodal rep- resentation with embedded visual guiding objects for named entity recognition social media posts. In Proceedings of the 28th ACM International Confer- ence on Multimedia, pages 1038-1046.
Jianfei Yu, Jing Jiang, Yang. and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal trans- former, Association for Computational Linguistics.
Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multi- modal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of AAAI conference on artificial intelligence, volume 35, pages 14355.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network named en- tity recognition in tweets. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Changmeng Zheng, Zhiwei Wu, Tao Wang. Yi Cai, and Qing 2020. Object-aware multimodal named entity recognition in social media posts with adver- sarial learning. IEEE Transactions Maltimedia, 23:2520-255


Reviews:
[{"review_id": "2q94oAWoT6", "reviewer": "Reviewer_YD3t", "summary": "", "questions": "1. Since ESPVR is jointly trained by ESI loss and MNER loss, it would be interesting to see qualitative/quantitative analysis about how ESPVR improve ESI task.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We feel great thanks for your professional review work on our article. Please allow us to address the key comments you raise:\n\n__Reasons To Reject:__\n\n1.There can be more details for implementation details. e.g. What is the lamda value at row 423? Is it learnable or manually selected? In this joint learning task, which loss plays a more important role for training?\n\nThe lamda value in line 423 is 0.08 and was chosen manually. Because the main task is modeled larger, the loss of the main task MNER is more useful for training in joint learning.\n\n__Questions For The Authors:__\n\n1.Since ESPVR is jointly trained by ESI loss and MNER loss, it would be interesting to see qualitative/quantitative analysis about how ESPVR improve ESI task.\n\nThe effect of the ESI task during joint training was as follows:\n\nTwitter-2017: 90.38(P), 92.38(R), 91.37(F1)\n\nTwitter-2015: 81.49(P), 82.97(R), 82.22(F1)\n\nFrom the experimental results, the effect of the auxiliary task ESI exceeded the effect of the primary task MNER, indicating that the auxiliary task was favorable to the primary task, and that the primary task did not impair the effect of the auxiliary task. We will add that part of the experiment and its analysis to the paper later.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper is clear.\n\n2. The proposed ESPVR model has a rich architecture that creatively integrates visual and textual information. The combination of BERT for text and Swin Transformer for visual representation is novel and could be promising.\n\n3. The idea is innovative and well-articulated.", "reasons_to_reject": "1. There can be more details for implementation details. e.g. What is the lamda value at row 423? Is it learnable or manually selected? In this joint learning task, which loss plays a more important role for training?", "questions_to_authors": "1. Since ESPVR is jointly trained by ESI loss and MNER loss, it would be interesting to see qualitative/quantitative analysis about how ESPVR improve ESI task.", "paper_topic_and_main_contributions": "The paper is motivated by the challenges and shortcomings of existing methods for acquiring visual information in the context of Multimodal Named Entity Recognition (MNER). Specifically, the authors identify two main drawbacks:\n\n1) Attention-based methods that extract visual regions related to the text are distracted by the entire image, rather than focusing on the most relevant visual regions.\n2) Object detection methods, such as Mask R-CNN, have a limited range of recognition categories and may produce visual regions that do not correspond to entities in the text.\n\nMotivated by the above challenges, this paper presents a novel model, ESPVR, for the task of Multimodal Named Entity Recognition (MNER), which extracts entities from text and classifies them with the assistance of image information. The ESPVR model consists of four major modules: Feature Extraction, Entity Spans Position Visual Regions (ESPVR), Multimodal Interaction, and CRF Decoding. It employs BERT for text encoding and Swin Transformer for visual representation\n\nExperiments show that the ESPVR achieved SOTA in both Twitter datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9rCZaFKZC5", "reviewer": "Reviewer_b5zS", "summary": "", "questions": "1. Upon the under-performance on Twitter 2015, please explain how would you move further to tackle such issue?\n\n2. Please offer sample evidences on the claimed issue existed in the dataset of Twitter 2015?\n\n3. Regarding the model design, do the authors posit that an alternative structure, such as a split-attention based network with segmented image subdomains. Do you think this would benefit or harm the model's performance? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable comments. Please allow us to address the key comments you raise:\n\n__Reasons To Reject:__\n\n1.The dataset used for experimentation lacks sufficient diversities, mainly single sourced on Twitter's data. However, it is possible that the inherent correlation between images posted on Twitter and their associated textual content results in a stronger coherence unique to the Twitter platform, compared to alternative data sources, such as E-commerce's data. Consequently, further evidence is needed to establish the extrapolation of the present work's scalability and efficacy to other domains.\n\nWe think this is an excellent suggestion. Therefore, we again looked at the datasets used in the previous work, and also consulted with experts in the field, and we found that the datasets currently available for English multimodal named entity recognition include Twitter-2015, Twitter-2017 and SNAP, where Twitter-2017 is the version after removing the illegal information from SNAP. Although we cannot experiment in other domains at this time, we experimented with the same datasets as the work mentioned in the comparison experiment. We will follow the development of the dataset and add relevant experiments for future work.\n\n2.A minor worry arises from the model's marginal enhancement and lack of progress on the Twitter 2015 dataset. While the authors briefly address reasons of solution's underperformance, they don't provide substantiating evidence. It's recommended to share their rationale and supporting data. Additionally, exploring other perspectives, like investigating potential disconnection between images and text in Twitter 2015, could provide valuable insights. All these would offer a clear understanding of the model's limitations for future studies.\n\nThe reasons why the model didn't work well on Twitter-2015 include two: many image-text pairs are irrelevant [1, 2, 3, 4], and many entities are mislabeled [5]. In fact, there are many such instances. We think that they have affected our MNER performance. We give relevant references, and a sample for each reason. Also, we will add the relevant samples to the final version of the paper.\n\n(1) Many image-text pairs are irrelevant.\n\nThe content of the text is \"Nice image of Kevin Love and Kye KorveDuring 1st half #NBAFinals #Cavsin9 #Cleveland\". Obviously, The text talks about basketball players, while the image depicts two bins standing in front of a wall.\n\n(2) Many entities are mislabeled.\n\nThe content of the text is \"Dr. Julie Hood, right, teacher w/Dade schools, w/fiance Marilyn. They're getting married in a few weeks.\". In the dataset, \"Julie Hood\" is labeled \"I-PER I-PER\", but the correct label for \"Julie Hood\" is \"B-PER I-PER\".  \n\n[1] Vempala A, Preoţiuc-Pietro D. Categorizing and inferring the relationship between the text and image of twitter posts[C]//Proceedings of the 57th annual meeting of the Association for Computational Linguistics. 2019: 2830-2840.\n\n[2] Yu J, Jiang J, Yang L, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer[C]. Association for Computational Linguistics, 2020.\n\n[3] Chen D, Li Z, Gu B, et al. Multimodal named entity recognition with image attributes and image knowledge[C]//Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 11–14, 2021, Proceedings, Part II 26. Springer International Publishing, 2021: 186-201.\n\n[4] Liu L, Wang M, Zhang M, et al. Uamner: uncertainty-aware multimodal named entity recognition in social media posts[J]. Applied Intelligence, 2022, 52(4): 4109-4125.\n\n[5] Liu P, Wang G, Li H, et al. Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Media[J]. arXiv preprint arXiv:2210.14163, 2022.\n\n__Questions For The Authors:__\n\n1.Upon the under-performance on Twitter 2015, please explain how would you move further to tackle such issue?\n\nIn future work, we will propose an effective method to solve the problem of image-text irrelevance, by filtering out text-irrelevant images before acquiring the local visual information, and acquiring the local visual region only from the text-relevant images. Since the problem of image-text irrelevance exists not only in Twitter 2015, but also has a small portion in Twitter 2017, we believe that this method will achieve better results on both datasets.\n\n2.Please offer sample evidences on the claimed issue existed in the dataset of Twitter 2015?\n\nThe answer to this question can be found in response to question 2 in \"Reasons To Reject\".\n\n3.Regarding the model design, do the authors posit that an alternative structure, such as a split-attention based network with segmented image subdomains. Do you think this would benefit or harm the model's performance?\n\nThe use of Swin Transformer is part of our method, and replacing it with a split-attention based network will have a little impact on our method, but not much. Swin Transformer can split the image into many patches, and eventually we can get the visual feature representation corresponding to each pacth. The ESPVR module consists of two modules: ESI module, VRP module. Firstly, the ESI module identifies all entity spans in the text, then the VRP module uses the entity spans to obtain entity features and uses the entity features to locate the visual regions. In the step of localizing the visual visual region, we do the computation through the entity features and the corresponding visual feature of the pacth, and select the patches that are most relevant to the entity to form the local visual information. In other words, Swin Transformer can get the visual feature of each patch, which is helpful for us to locate the local visual information that is most relevant to the entities in the text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "First, the present work is well presented, demonstrating a clear and good motivation to address a prevailing challenge within the field. The lack of identified visual regions for NER stands as a long-existed issue in the domain. The solution proposed by authors offers a remedy to this issue. The innovative proposition of identifying the most relevant visual features based on textual cues not only shows authors' novelty but also lays a new direction for future studies.\n\nSecondly, this work offers solid mathematical justification and derivation for the methodology and good explanation on the methodology. This improves the reproducibility of this work. \n\nThirdly, this study is substantiated by a comprehensive set of experiments, including good ablation studies. The results, presented both qualitatively and quantitatively, justifies the proposed model's theoretical and practical values.", "reasons_to_reject": "The dataset used for experimentation lacks sufficient diversities, mainly single sourced on Twitter's data. However, it is possible that the inherent correlation between images posted on Twitter and their associated textual content results in a stronger coherence unique to the Twitter platform, compared to alternative data sources, such as E-commerce's data. Consequently, further evidence is needed to establish the extrapolation of the present work's scalability and efficacy to other domains.\n\nA minor worry arises from the model's marginal enhancement and lack of progress on the Twitter 2015 dataset. While the authors briefly address reasons of solution's underperformance, they don't provide substantiating evidence. It's recommended to share their rationale and supporting data. Additionally, exploring other perspectives, like investigating potential disconnection between images and text in Twitter 2015, could provide valuable insights. All these would offer a clear understanding of the model's limitations for future studies. ", "questions_to_authors": "1. Upon the under-performance on Twitter 2015, please explain how would you move further to tackle such issue?\n\n2. Please offer sample evidences on the claimed issue existed in the dataset of Twitter 2015?\n\n3. Regarding the model design, do the authors posit that an alternative structure, such as a split-attention based network with segmented image subdomains. Do you think this would benefit or harm the model's performance? ", "paper_topic_and_main_contributions": "This work introduces a novel technique for named entity recognition (NER) that combines image and text modalities. The focal point of this work is the introduction of a novel learning algorithm module named \"Entity Spans Position Visual Regions\" (ESPVR), designed to address common challenges encountered in existing solutions. The authors claim that the ESPVR module enhances the efficacy of multimodal named entity recognition by identifying the most relevant visual regions corresponding to entities within the text. This information is subsequently utilized to enhance the attention layer, facilitating improved information concentration. In contrast to extant methodologies relying on the entire image feature map, the proposed ESPVR module offers a more targeted and effective approach. The research methodology is substantiated by a well-designed experimental setup, featuring both quantitative and qualitative assessments. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0ZxOjvCcNX", "reviewer": "Reviewer_mY78", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your valuable comments, which will serve to improve the paper considerably. Please allow us to address the key comments you raise:\n\n1.The reported experimental results cannot strongly demonstrate the effectiveness of the proposed method.\n\n（1）The main reason for this problem is due to the presence of many irrelevant image-text pairs in the Twitter dataset [1, 2, 3, 4], especially Twitter-2015. The contribution of this paper is to firstly discover two problems with existing methods for acquiring local visual information, and propose a method. Although the effect of our proposed method does not achieve SOTA on all evaluation metrics, we still significantly outperform other works in some important metrics. This is enough to show that the motivation of the research in this paper is correct, and that the method we propose is effective. We do believe this can advance the growth of MNER.\n\nThe main limitation of this paper is that it does not deal with the problem of image-text irrelevance. In our future work, we will propose an effective method to solve the problem of image-text irrelevance , by filtering out text-irrelevant images before acquiring the local visual information, and acquiring the local visual region only from the text-related images, in order to further improve the modeling effect.\n\n（2）We reran the \"Twitter-2017→Twitter-2015\" cross-domain experiment and the results remain the same as documented in the paper, where the evaluation metrics are also correct. To further prove the authenticity of the experiments, before 31 Aug 2023, 23:59 China Standard Time，we will put the code and data about the \"Twitter-2017→Twitter-2015\" and \"Twitter-2015→Twitter-2017\" experiments on github, and named it ESPVR.\n\n2.In line 284, a transformer layer with self-attention is used to capture the intra-modality relation for the test modality. However, there're a lot of self-attention transformer layers in BERT. Why not using the attention scores in the last self-attention transformer layer?\n\nThe self-attention effect is related to the number of layers of the transformer used, and this paper achieves the best self-attention effect with the addition of these 4 layers of the transformer.\n\n3.In line 322, softmmax -> softmax\n\nWe apologize greatly for our carelessness. We have carefully checked the entire text and there are no spelling errors at this time.\n\n4.Will the coordination of  $b_d$ exceed the scope of the patches?\n\nThis does not happen. When we wrote the algorithm, we took the out-of-bounds problem into account. The coordinates $d$ of $b_d$ are obtained by adding $\\alpha$ to the horizontal and vertical coordinates of the coordinate $t$, respectively. If the horizontal coordinate of coordinate $t$ crosses the line after adding $\\alpha$, the rightmost coordinate is used as the horizontal coordinate of $d$. If the vertical coordinate of coordinate $t$ crosses the line after adding $\\alpha$, the bottom coordinate is used as the horizontal coordinate of $d$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and is easy to follow.", "reasons_to_reject": "- The reported experimental results cannot strongly demonstrate the effectiveness of the proposed method. \n  - In Table 1, for the proposed method, only 6 of the total 14 evaluation metrics achieve SOTA performances. \n  - In Table 2, for the proposed method, only 8 of the total 14 evaluation metrics achieve SOTA performances. In addition, under the setting of \"Twitter-2017 $\\rightarrow$ Twitter-2015\", why the proposed method achieves best overall F1, while not achieves best F1 in all single types?\n  - In Table 3, for the proposed method, 9 of the total 14 evaluation metrics achieve SOTA performances, which means that when ablating some modules, the performance of the proposed method will improve. Furthermore,  The performance improvement that adding a certain module can bring is not obvious.\n- In line 284, a transformer layer with self-attention is used to capture the intra-modality relation for the test modality. However, there're a lot of self-attention transformer layers in BERT. Why not using the attention scores in the last self-attention transformer layer?\n- In line 322, softmmax -> softmax\n- Will the coordination of $b_d$ exceed the scope of the patches?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the task of MNER.\nTo solve the two drawbacks existing in previous methods, this paper proposes a novel method, called ESPVR.\nThe experiments on two datasets demonstrate the effectiveness of the proposed method. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NfeWR7vxgQ", "reviewer": "Reviewer_Xr7Z", "summary": "", "questions": "Please refer to Reasons to reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your efforts in improving the quality of our paper. Please allow us to address the key comments you raise:\n\n1.The motivation behind the paper is not entirely convincing, and it would be better to provide some practical examples to support the argument. Additionally, the author points out a flaw in current MNER models, stating that attention mechanisms tend to be dispersed across the entire image, making it difficult to focus on the most relevant visual regions for the text. However, it is worth noting that the proposed VRP module is also based on attention mechanisms, which presents a contradiction between the two arguments.\n\n（1）We think this proposal is very valuable. We have found examples that support our motivation and will add to the final version of the paper. Since examples require the use of images, but here it is not possible to insert images, we are very sorry that we are not able to describe examples here.\n\n（2）Our approach doesn't contradict that viewpoint. The attention mechanism itself is not problematic and is a very good method. Although both use the attention mechanism, the local visual information we acquire is the most relevant to the entities in the text. When previous work used the attention mechanism to acquire visual information, there was no special consideration for the fact that attention would be distracted by the entire image, resulting in previous work acquiring visual information that was not only relevant to the text, but also contained a large amount of irrelevant visual information, which ultimately impacted the prediction. The essence of our method is to calculate the attention distribution of entities in the text and visual regions in the image, discard the part of the visual regions with low relevance scores through the VRP module, and keep the part of the visual regions that are most relevant to the entities in the text as the local visual information, which solves the problem of a large amount of irrelevant visual information due to the distraction problem of the attention mechanism. \n\n2.It would be more persuasive to include a visualization experiment to demonstrate the effectiveness of the proposed ESPVR module in capturing the most relevant visual parts for text entities. The current experimental design does not provide an intuitive way to showcase this.\n\nWe think this proposal is very valuable. We have obtained the visualization results and will add them to the final version of the paper. We are very sorry that we are unable to show the visualization results here, since the visualization requires the use of images, but they cannot be inserted here.\n\n3.I believe that this paper is not yet ready for submission. It is important to adhere to the basic standards of formal English writing and ensure that the paper is well-organized. Additionally, Figure 1 has a low resolution and contains errors, such as the misspelling of \"Multimodal Inerrection Module,\" which should be corrected to \"Multimodal Interaction Module.\"\n\nWe apologize profusely for our carelessness. We have invited professionals to polish the language in this paper. Moreover, we have carefully checked the whole text , and now there are no spelling mistakes, and we have modified Figure 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The effectiveness of the proposed method is evaluated across diverse tasks, including self-domain and cross-domain scenarios, yielding promising results.", "reasons_to_reject": "1. The motivation behind the paper is not entirely convincing, and it would be better to provide some practical examples to support the argument. Additionally, the author points out a flaw in current MNER models, stating that attention mechanisms tend to be dispersed across the entire image, making it difficult to focus on the most relevant visual regions for the text. However, it is worth noting that the proposed VRP module is also based on attention mechanisms, which presents a contradiction between the two arguments.\n\n2. It would be more persuasive to include a visualization experiment to demonstrate the effectiveness of the proposed ESPVR module in capturing the most relevant visual parts for text entities. The current experimental design does not provide an intuitive way to showcase this.\n\n3. I believe that this paper is not yet ready for submission. It is important to adhere to the basic standards of formal English writing and ensure that the paper is well-organized. Additionally, Figure 1 has a low resolution and contains errors, such as the misspelling of \"Multimodal Inerrection Module,\" which should be corrected to \"Multimodal Interaction Module.\"", "questions_to_authors": "Please refer to Reasons to reject.", "paper_topic_and_main_contributions": "The primary objective of this paper is to explore multimodal named entity recognition, and to propose a novel approach called Entity Spans Position Visual Regions (ESPVR) that effectively captures the most relevant visual regions for entities in text. Specifically, the ESPVR module is comprised of two key components: the Entity Span Identification (ESI) module and the Visual Regions Positioning (VRP) module. The experimental results demonstrate that the proposed method outperforms current state-of-the-art techniques on the Twitter-2017 dataset, and yields competitive results on Twitter-2015.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces a novel technique for NER augmented with the visual modality, and introduces a novel learning algorithm module named Entity Spans Position Visual Regions (ESPVR), designed to address common challenges encountered in existing solutions. The authors claim that the ESPVR module enhances the efficacy of multimodal named entity recognition by identifying the most relevant visual regions corresponding to entities within the text. The reviewers generally agree on the novelty of the work and soundness of the presented method. There has been minor concern on missing experimental setup details, persuasiveness of some results, and sufficient diversity in experimental datasets.
Decision: Accept-Findings