Grounded Multimodal Named Entity Recognition on Social Media
Year: 2023

Abstract:
In recent years, Multimodal Named Entity Recognition (MNER) on social media has attracted considerable attention. However, existing MNER studies only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction and insufficient for entity disambiguation. To solve these issues, in this work, we introduce a Grounded Multimodal Named Entity Recognition (GMNER) task. Given a text-image social post, GMNER aims to identify the named entities in text, their entity types, and their bounding box groundings in image (i.e., visual regions). To tackle the GMNER task, we construct a Twitter dataset based on two existing MNER datasets. Moreover, we extend four well-known MNER methods to establish a number of baseline systems and further propose a Hierarchical Index generation framework named H-Index, which generates the entity-type-region triples in a hierarchical manner with a sequence-tosequence model. Experiment results on our annotated dataset demonstrate the superiority of our H-Index framework over baseline systems on the GMNER task. Our dataset annotation and source code are publicly released at https://github.com/NUSTM/GMNER.

heading: 1 Introduction
text: Fueled by the rise of phones and tablets with camera functions, user posts on social media platforms such as Twitter are increasingly multimodal, e.g., containing images in addition to text. The explosive growth of multimodal posts is far beyond humans’ capability to digest them. Hence, it presents a pressing need for automatically extracting important information such as entities and relations from the large amount of multimodal posts, which is crucial for structured knowledge graph construction to help people efficiently understand massive
∗Equal contribution. † Corresponding author.
content. As an emerging subtask for multimodal knowledge graph construction (Liu et al., 2019), Multimodal Named Entity Recognition (MNER) on social media has recently attracted increasing attention (Zhang et al., 2018; Moon et al., 2018). Given a text-image social post, MNER aims to recognize named entities in text and classify them into pre-defined types such as person (PER), location (LOC), and organization (ORG).
Most previous studies formulate the MNER task as a sequence labeling problem, which focus on (1) designing effective attention mechanisms to model the vision-language interaction to obtain vision-aware word representations (Lu et al., 2018; Yu et al., 2020; Zhang et al., 2021a; Chen et al., 2022b) or (2) converting the images into the textual space by generating image captions and object tags (Chen et al., 2021b; Wang et al., 2022a). Inspired by the success of applying the machine reading comprehension (MRC) framework in NER (Li et al., 2020b), several recent works formalize the MNER task as a MRC problem, which extract entities by answering queries about entity types (Jia et al., 2022, 2023).
However, existing MNER studies mainly regard the visual features as additional clues to help enhance the performance of the text-only NER task, which suffer from several limitations. First, as shown in Fig. 1, previous MNER works only ex-
9141
tract entity-type pairs in text, but failing to link the entities to their corresponding bounding boxes in image. The extracted entity-type pairs are solely useful for constructing text-only knowledge graph rather than multimodal knowledge graph. Moreover, only identifying entity-type pairs in text is often insufficient for entity disambiguation. For example, in Fig. 1, without the grounded yellow bounding box, it is hard to infer the (Michael Jordan, PER) pair refers to the professor in UC Berkeley rather than the famous basketball player.
To address these issues, in this paper, we propose a new task named Grounded Multimodal Named Entity Recognition (GMNER), aiming to extract the named entities in text, their entity types, and their bounding box groundings in image. Given the example in Fig. 1, the goal is to extract three entitytype-region multimodal triples, i.e., (Michael Jordan, PER, yellow box), (the Fields Institute, ORG, blue box) and (Toronto, LOC, None). GMNER presents the following challenges: (1) apart from extracting entity-type pairs, it requires predicting whether each entity has a grounded region in image; (2) for entities with visually grounded regions, it needs to locate its corresponding bounding box groundings (i.e., visual regions).
To tackle the GMNER task, we first construct a Twitter dataset based on two benchmark MNER datasets, in which we manually annotate the bounding box groundings for each entity-type pair labeled by the two datasets. With the new dataset, we benchmark the GMNER task by establishing a number of baseline systems based on four wellknown MNER methods. Furthermore, inspired by the success of the index generation framework in the NER task (Yan et al., 2021), we formulate the GMNER task as a multimodal index generation problem by linearizing all the entity-type-region triples into a position index sequence. We then propose a Hierarchical Index generation framework named H-Index, aiming to address the aforementioned two challenges of GMNER in a hierarchical manner. Specifically, a pre-trained sequence-tosequence model BART (Lewis et al., 2020) is first employed to encode the textual and visual inputs to generate a set of triples, which contain the indexes of entity positions, entity types, and groundable or ungroundable indicators. Moreover, for groundable entities, we further stack a visual output layer to predict the distribution over candidate visual regions for entity grounding.
The main contributions of our work can be summarized as follows:
• We introduce a new task named Grounded Multimodal Named Entity Recognition (GMNER), which aims to extract all the entity-type-region triples from a text-image pair. Moreover, we construct a Twitter dataset for the task based on two existing MNER datasets.
• We extend four well-known MNER methods to benchmark the GMNER task and further propose a Hierarchical Index generation framework named H-Index, which generates the entity-typeregion triples in a hierarchical manner.
• Experimental results on our annotated dataset show that the proposed H-Index framework performs significantly better than a number of unimodal and multimodal baseline systems on the GMNER task, and outperforms the second best system by 3.96% absolute percentage points on F1 score.

heading: 2 Task Formulation
text: Given a multimodal input containing a piece of text with n words s = (s1, . . . , sn) and an accompanying image v, the goal of our Grounded Multimodal Named Entity Recognition (GMNER) task is to extract a set of multimodal entity triples:
Y = {(e1, t1, r1), . . . , (em, tm, rm)}, (1)
where (ei, ti, ri) denotes the i-th triple, ei is one of the entities in text, ti refers to the type of ei which belongs to four pre-defined entity types including PER, LOC, ORG, and MISC, and ri denotes the visually grounded region of entity ei. It is worth noting that if there is no grounded region of entity ei, ri is None; otherwise, ri consists of a 4-D spatial feature containing the top-left and bottomright positions of the grounded bounding box, i.e., (rx1i , r y1 i , r x2 i , r y2 i ).

heading: 3 Dataset
text: Since there was no available corpus for the GMNER task, we construct a Twitter dataset as follows.
Data Collection. Our dataset is built on two benchmark MNER datasets, i.e., Twitter-15 (Zhang et al., 2018) and Twitter-17 (Yu et al., 2020), which have already annotated all the entities and their types for each multimodal tweet. To alleviate the annotation difficulty, we filter samples with missing images or with more than 3 entities belonging to
the same type, and then merge the remaining 12K+ samples as our raw dataset for annotation.
Bounding Box Annotation. We employ three graduate students to independently annotate the grounded regions (i.e., bounding boxes) for each labeled entity based on a widely-used image annotation tool named LabelImg1. Fleiss Kappa (Fleiss, 1971) is adopted to measure the annotation agreement. Note that if the Intersection over Union (IoU) score between two annotations is larger than 0.5, we regard them as consistent annotations. The Fleiss score between three annotators is K = 0.84, indicating a substantial annotation agreement. To ensure the quality of our dataset, we remove samples in which the IoU score between annotations is less than 0.5. Finally, we obtain 10,159 samples and randomly select 10K samples as our TwitterGMNER dataset, followed by averaging the three annotations as the ground-truth bounding box annotation for each sample.
Dataset Analysis. Following Moon et al. (2018), we divide our dataset into train (70%), validation (15%), and test sets (15%). As shown in Table 1, our dataset contains 16,778 entities and around 60% entities do not have a grounded bounding box. For the remaining 6,716 groundable entities, we manually annotate a total of 8,090 bounding boxes, which indicates that each entity may correspond to more than one bounding box.
In Table 2, we compare our dataset with six NER datasets for social media. WNUT16 (Strauss et al., 2016) and WNUT17 (Derczynski et al., 2017) are two text-only NER datasets released at the 2nd and 3rd Workshop on Noisy User-generated Text. Twitter-Snap, Twitter-15, and Twitter-17 are three benchmark MNER datasets released by Lu et al. (2018), Zhang et al. (2018), and Yu et al. (2020), respectively. WikiDiverse is a new dataset introduced by Wang et al. (2022b). Compared with existing datasets, our dataset contains more annotated samples (i.e., 10K) and is the first dataset containing both textual and visual annotations.
Fig. 2 (left) shows the distribution of the number 1https://github.com/tzutalin/labelImg
of bounding boxes in each sample. We can observe that the image in 44.2% samples is unrelated to any entity mentioned in text, whereas 41.8% samples only contain one bounding box and around 14.0% samples contain two or more bounding boxes. This indicates the necessity and challenge of achieving text-image and entity-image alignments for GMNER. In Fig. 2 (right), we show that most entities with the PER type are grounded in the image, whereas entities with the other three types (especially LOC) are usually not grounded in the image.

heading: 4 Methodology
text: In this section, we present the details of our proposed Hierarchical Index generation (H-Index) framework.
Overview. As illustrated in Fig. 3, H-Index formulates the GMNER task as an index generation problem, which resorts to a pre-trained sequenceto-sequence model BART (Lewis et al., 2020) to encode the textual and visual inputs, followed by decoding the indexes of entities, types, and groundable or ungroundable indicators. For entities with groundable indicators, another output layer is added to predict the distribution over visual regions for entity grounding.

heading: 4.1 Feature Extraction
text: Text Representation. Given the input text s = (s1, . . . , sn), we feed it to the embedding matrix
of BART to obtain the text representation as T = {e1, . . . , en}, where ei ∈ Rd.
Visual Representation. Given the input image v, we employ a widely-adopted object detection method VinVL (Zhang et al., 2021b) to identify all the candidate objects (i.e., visual regions). After ranking these objects based on their detection probabilities, we keep the top-K objects and extract the mean-pooled convolutional features from VinVL to obtain fixed-size embeddings for visual regions, denoted by R = {r1, . . . , rK}, where ri ∈ R2048 is the representation of the i-th region. We then use a linear layer to transform R into the same dimension of text, and thus the regional representation is denoted as V = {v1, . . . ,vK}, where vi ∈ Rd.

heading: 4.2 Design of Multimodal Indexes
text: As mentioned before, the GMNER task requires extracting three kinds of information, including entity mentions in text, entity types, and visual regions in image. To project these different information into the same output space, we draw inspiration from the NER task (Yan et al., 2021) and use unified position indexes to pointing to these information.
Specifically, we can infer from Table 1 that around 60% entities do not have grounded visual regions in image, which indicates the correct prediction of the relevance between entities and images is essential to entity grounding. Thus, we first transform the entity-image relation into two indexes (i.e., 1 and 2 in the left of Fig. 3) to indicate
whether each entity is groundable or ungroundable. Next, we use four indexes (i.e., 3 to 6) to refer to four entity types. Because the input text s is a sequence with n words, we directly use n position indexes starting from 7 to refer to each word in s.
For example, given the textual and visual inputs in Fig. 3, its output index sequence contains three entity-relation-type triples. The first triple [7,8,1,3] refers to {Michael Jordan, groundable, PER}, the second triple [12,2,4] denotes {Toronto, ungroundable, LOC}, and the third triple [19,20,21,1,5] refers to {the Fields Institute, groundable, ORG}. Formally, let us use y to denote the output index sequence.

heading: 4.3 Index Generation Framework
text: Given a multimodal input, we employ a sequenceto-sequence model BART (Lewis et al., 2020) to generate the output index sequence y.
Encoder. We first feed the concatenation of text and visual representations to the BART encoder to obtain the hidden representation as follows:
He = [HeT;H e V] = Encoder([T;V]), (2)
where HeT ∈ Rn×d and HeV ∈ RK×d are textual and visual parts of He ∈ R(n+K)×d, respectively.
Decoder. At the i-th time step, the decoder takes He and the previous decoder output y<i as inputs
to predict the output probability distribution p(yi):
hi = Decoder(He;y<i), (3) H̄eT = ( T+ MLP(HeT) ) /2, (4) p(yi) = Softmax([C; H̄ e T] · hi), (5)
where MLP refers to a multi-layer perceptron, C = TokenEmbed(c) refers to the embeddings of two indicator indexes, four entity type indexes, and special tokens such as the “end of sentence” token 
, and · denotes the dot product.
The cross entropy loss is used to optimize the parameters of the generative model as follows:
LT = − 1 NM
N∑
j=1
M∑
i=1
log p(yji ), (6)
where N and M refer to the number of samples and the length of output index sequence, respectively.

heading: 4.4 Entity Grounding
text: Lastly, for groundable entities, we further stack another output layer to perform entity grounding.
Specifically, let us use hk to refer to the time step whose predicted index is the groundable indicator (i.e., index 1). We then obtain the probability distribution over all the visual regions from VinVL, denoted by p(zk) as follows:
H̄eV = ( V + MLP(HeV) ) /2, (7)
p(zk) = Softmax(H̄eV · hk). (8) Region Supervision. As shown in the top of Fig. 3, since the visual regions from VinVL are different from the ground-truth (GT) bounding boxes, we first compute the overlap between visual regions and GT bounding boxes based on their Intersection over Union (IoU) scores. Note that for each visual region, we compute its IoU scores with respect to all GT bounding boxes of a given entity and take the maximum value as its IoU score. Moreover, for visual regions whose IoU score is less than 0.5, we follow the practice in visual grounding (Yu et al., 2018b) by setting its IoU score as 0. Next, we renormalize the IoU score distribution as the region supervision for a given entity, denoted by g(zk).
The objective function of entity grounding is to minimize the Kullback-Leibler Divergence (KLD) loss between the predicted region distribution p(zk) and the region supervision g(zk)：
LV = 1 NE
N∑
j=1
E∑
k=1
g(zjk) log g(zjk)
p(zjk) , (9)
Algorithm 1 Our Entity-Groundable/UngrounableType Triple Recovery Algorithm
Input: Predicted sequence ŷ = [ŷ1, ..., ŷl] and ŷl ∈ [1, n + |c|], where c is the list of two indicator indexes, four entity type indexes, and special tokens. Output: Triples E 1: E = {}, e = [], i = 1 2: while i <= l do 3: yi = Y [i] 4: if yi < |c| then 5: if len(e) > 0 then 6: if indexes in e is ascending then 7: if yi = 1 or yi = 2 then 8: E.add(e, cyi , cyi+1) 9: end if 10: end if 11: end if 12: e = [] 13: i = i+ 2 14: else 15: e.append(yi) 16: end if 17: i = i+ 1 18: end while 19: return E
where E is the number of groundable entities. In the training stage, we combine LT and LV as the final loss of our H-Index model:
L = LT + LV . (10)

heading: 4.5 Entity-Type-Region Triple Recovery
text: In the inference stage, given a multimodal input, we use the trained H-Index model to generate the index sequence ŷ in an autoregressive manner based on greedy search, and predict the region distribution p(ẑk) for the k-th groundable entity.
With the output index sequence, we can first convert each index to its original meaning and then recover (entity, groundable/ungroundable, type) triples based on the index span of each element. The full algorithm is shown in Algorithm 1.
For the j-th ungroundable entity, the predicted triple is (ej , tj ,None). For the k-th groundable entity, we regard the visual region with the highest probability in p(ẑk) as the predicted bounding box, and take its 4-D coordinates rk = (x1k, y 1 k, x 2 k, y 2 k) as the visual output. Thus, the predicted triple of the k-th groundable entity is (ek, tk, rk).

heading: 5 Experiments

heading: 5.1 Experimental Settings
text: For our proposed framework H-Index, we employ the pre-trained VinVL model released by (Zhang et al., 2021b) to detect top-K visual regions, and use the pre-trained BARTbase model from (Lewis et al., 2020) to initialize the parameters in the index generation framework in Section 4.3. Hence, the hidden dimension d is set to the default setting 768. The batch size and training epoch are set to 32 and 30, respectively. During training, we use the AdamW optimizer for parameter tuning. For the learning rate and the number of candidate visual regions K, we set their values to 3e-5 and 18 after a grid search over the combinations of [1e-5, 1e-4] and [2, 20] on the development set.
Evaluation Metrics. The GMNER task involves three elements, i.e., entity, type, and visual region. For entity and type, we follow previous MNER works to use the exact match for evaluation (Zhang et al., 2018). For visual region, if it is ungroundable, the prediction is considered as correct only when it is None; otherwise, the prediction is considered as correct only when the IoU score between the predicted visual region and one of the groundtruth (GT) bounding boxes is large than 0.5 (Mao et al., 2016). The correctness of each element is computed as follows:
Ce/Ct = { 1, pe/pt = ge/gt; 0, otherwise.
(11)
Cr =    1, pr = gr = None; 1, max(IoU1, ..., IoUj) > 0.5; 0, otherwise. (12)
where Ce, Ct, and Cr denote the correctness of entity, type, and region predictions, pe, pt, and pr denote the predicted entity, type, and region, ge, gt, and gr denote the gold entity, type, and region, and IoUj denotes the IoU score between the predicted region pr with the j-th GT bounding box gr,j . We then calculate precision (Pre.), recall (Rec.), and F1 score to measure the performance of GMNER:
correct = { 1, Ce and Ct and Cr; 0, otherwise.
(13)
Pre = #correct #predict , Rec = #correct #gold , (14) F1 = 2× Pre×Rec Pre+Rec , (15)
where #correct denotes the number of predicted triples that match the gold triples, and #predict and #gold are the number of predicted and gold triples.

heading: 5.2 Baseline Systems
text: Since GMNER is a new task and there is no existing method for comparison, we first consider several text-only methods as follows:
• HBiLSTM-CRF-None, which uses the hierarchical BiLSTM-CRF model (Lu et al., 2018) to extract entity-type pairs, followed by setting the region prediction to the majority class, i.e., None;
• BERT-None, BERT-CRF-None, and BARTNERNone, which replace the hierarchical BiLSTMCRF model in HBiLSTM-CRF-None with BERT (Devlin et al., 2019), BERT-CRF, and BARTNER (Yan et al., 2021), respectively.
Moreover, we develop a pipeline approach as a strong baseline, which first uses any previous MNER method to extract entity-type pairs and then predicts the bounding box for each pair with an Entity-aware Visual Grounding (EVG) model.
Specifically, given the i-th extracted entity-type pair (ei, ti) from existing MNER methods as well as the textual input s, we construct the textual input as follows: [[CLS], s, [SEP], ei, [SEP], ti, [SEP]], which is fed to a pre-trained BERTbase model (Devlin et al., 2019) to obtain the text representation T. We then use the feature extraction method in Section 4.1 to obtain the visual representation V. Next, a Cross-Model Transformer layer (Tsai et al., 2019) is utilized to model the interaction between the text and visual representations as follows: H = CMT(V,T,T), where V and T are regarded as queries and keys/values, and H = {h1, . . . ,hK} is the generated hidden representation. For each visual region hj ∈ Rd, we add an output layer to predict whether it is the grounded region of (ei, ti): p(yj) = sigmoid(w⊤hj), where w ∈ Rd is the weight matrix. During inference, we choose the visual region with the highest probability. If the probability is higher than a tuned threshold, it implies the input entity-type pair is groundable and the predicted region is the top visual region; otherwise, the prediction region is None.
As shown in Table 3, we stack the EVG model over four well-known MNER methods as follows:
• GVATT-RCNN-EVG, which uses GVATT (Lu et al., 2018), a visual attention method based on
the BiLSTM-CRF model (Lample et al., 2016), to extract entity-type pairs, followed by applying the EVG model based on the objects detected by Faster R-CNN (Anderson et al., 2018);
• UMT-RCNN-EVG, which replaces the MNER method in GVATT-RCNN-EVG with UMT (Yu et al., 2020), a Multimodal Transformer approach with an auxiliary entity span detection task. UMT-VinVL-EVG is a variant of UMTRCNN-EVG, which replaces Faster R-CNN with VinVL;
• UMGF-VinVL-EVG is a variant of UMT-VinVLEVG using UMGF (Zhang et al., 2021a) for MNER, which models text-image interactions with a multimodal graph fusion network;
• ITA-VinVL-EVG is another variant of UMTVinVL-EVG using ITA (Wang et al., 2022a) for MNER, which translates images to captions and object tags, followed by sequence labeling.
• BARTMNER-VinVL-EVG is a variant of our HIndex approach, which first uses the index generation framework in Section 4.3 to identify entitytype pairs, and then uses the EVG model to predict the grounded bounding box for each pair.

heading: 5.3 Main Results
text: In Table 3, we show the results of different methods on the GMNER task. To better compare these methods, we also report the F1 score of two subtasks of GMNER, including MNER and Entity Extraction & Grounding (EEG). Note that MNER aims to identify the entity-type pairs whereas EEG aims to extract the entity-region pairs.
Results on GMNER. First, for text-based methods, it is clear that BARTNER-None significantly outperforms the other methods, which shows the effectiveness of the index generation framework
and agrees with the observation in existing NER works (Yan et al., 2021). Second, all the multimodal approaches consistently perform much better than text-based methods. This indicates the usefulness of our proposed Entity-aware Visual Grounding (EVG) baseline. Third, comparing all the multimodal baseline systems, BARTMNERVinVL-EVG obtains the best result, primarily due to its outstanding performance on the MNER subtask. Finally, we can clearly observe that our proposed H-Index framework outperforms the best baseline BARTMNER-VinVL-EVG by 3.96 absolute percentage points based on F1 score. The main reason for the performance gain is that all the baseline systems extract entity-types pairs followed by visual grounding, which suffer from the common error propagation issue of pipeline methods. In contrast, H-Index uses the unified index generation framework to directly generate the entity-type-region triples with a sequence-to-sequence model.
Results on MNER and EEG. First, for the MNER subtask, we can see that our H-Index framework performs generally better than most baselines but worse than BARTMNER-VinVL-EVG. We conjecture the reason is that all the baselines are pipeline methods and should achieve the best performance on each stage, whereas our H-Index model is an end-to-end approach for entity-typeregion triple extraction, which may only obtain the sub-optimal model on the MNER task. In addition, for the EEG subtask, H-Index significantly outperforms the best baseline by 5.44 absolute percentage points. These observations verify the effectiveness of our H-Index framework.

heading: 5.4 In-Depth Analysis
text: Ablation Study. In Table 5, we conduct ablation study of our H-Index framework. First, we replace
the KLD loss in Equation (9) with the cross-entropy loss. We find that the performance slightly drops, indicating that the KLD loss can better capture the relations between different visual regions for region detection. Moreover, we remove the hierarchical prediction of the groundable/ungroundable indicator in Section 4.3 and entity grounding in Section 4.4. Specifically, we use a special token to indicate whether the current time step in the decoder is for visual grounding, and then add a binary classification layer for each visual region, which is the same as the EVG baseline. As shown in Table 5, removing the hierarchical prediction leads to a performance drop of 2.09 percentage points.
Sensitivity Analysis of K. We use our H-Index model and BARTMNER-VinVL-EVG to analyze the impact of the number of object regions from VinVL on GMNER and EEG tasks. In Fig. 4, we can find the two methods gradually perform better as K becomes larger. This is because when K is small, the top-K regions from VinVL may not cover groundtruth visual regions. When K equals to 18, the two methods consistently achieve the best performance.

heading: 5.5 Case Study
text: We further conduct case study to compare the predictions of UMT-RCNN-EVG, BARTMNER-VinVLEVG, and H-Index on two test samples in our dataset. In Table 4.a, we find that all the methods correctly identify the three entity-type pairs. However, UMT-RCNN-EVG is confused with the two PER entities and wrongly predicts their grounded regions, while BARTMNER-VinVL-EVG fails to
identify the grounded region of Rose Byrne. In contrast, our H-Index model correctly identifies the bounding box groundings for the two PER entities. Similarly, in Table 4.b, the two baselines fail to identify either the correct entity type or the correct bounding box of Loki, whereas H-Index correctly grounds Loki onto the visual region with the dog, and predicts its type as MISC.

heading: 6 Related Work
text: NER on Social Media. Many supervised learning methods have achieved satisfactory results on formal text (Li et al., 2020a), including feature engineering methods (Finkel et al., 2005; Ratinov and Roth, 2009) and deep learning methods (Chiu and Nichols, 2016; Ma and Hovy, 2016). However, most of them perform poorly on social media, because the text on social media is often informal and short. To handle this problem, many social text-based features such as hashtags (Gimpel et al., 2010) and freebase dictionary (Ritter et al., 2011) are designed to enhance the performance of both feature-based methods (Baldwin et al., 2015) and deep learning methods (Limsopatham and Collier, 2016; Gerguis et al., 2016; Lin et al., 2017; Suman et al., 2021).
Multimodal NER on Social Media. With the
rapid growth of multimodal posts on social media, MNER has recently attracted much attention. Most existing MNER methods focus on modeling the text-image interactions by designing various kinds of cross-modal attention mechanism (Moon et al., 2018; Lu et al., 2018; Zheng et al., 2020). With the recent advancement of deep learning techniques, many studies focus on designing different neural networks for MNER, including Transformerbased methods (Sun et al., 2021; Xu et al., 2022; Chen et al., 2022a; Jia et al., 2023), Graph Neural Network-based methods (Zhang et al., 2021a; Zhao et al., 2022), Modality Translation-based methods (Chen et al., 2021b; Wang et al., 2022a), and Prompt-based models (Wang et al., 2022c). Despite obtaining promising results, these methods solely utilize the visual clues to better extract entity-type pairs. In contrast, the goal of our work is to extract entity-type-region triples from each multimodal post.
Visual Grounding. Given a natural language query, Visual Grounding (VG) aims to locate the most relevant object or region in an image. Most existing works on VG belong to two categories, i.e., one-stage methods and two-stage methods. The former focuses on utilizing recent end-to-end object detection models such as YOLO (Redmon and Farhadi, 2018) and DETR (Carion et al., 2020) to directly predict the visual region (Yang et al., 2019; Deng et al., 2021; Ye et al., 2022). The latter aims to first leverage object detection models (Ren et al., 2015; Zhang et al., 2021b) to obtain region proposals and then rank them based on their relevance to the text query (Yu et al., 2018a; Yang et al., 2020; Chen et al., 2021a). Our work follows the latter line of methods, which detects candidate visual regions with VinVL, followed by entity grounding.

heading: 7 Conclusion
text: In this paper, we introduced a new task named Grounded Multimodal Named Entity Recognition (GMNER), aiming to identify the named entities, their entity types, and their grounded bounding boxes in a text-image social post. Moreover, we constructed a new Twitter dataset for the task, and then extended four previous MNER methods to benchmark the task. We further proposed a Hierarchical Index generation framework (H-Index), which generates the entity-type-region triples in a hierarchical manner. Experimental results demonstrate the effectiveness of our H-index framework.
Limitations
Although we introduce a new GMNER task and propose a number of baseline systems and an HIndex framework, there are still some limitations in this work.
First, our GMNER task only requires identifying the visual regions that are correspondent to named entities mentioned in text. However, for each image, many visual regions may contain real-world entities that are not mentioned in text. Therefore, it would be interesting to further annotate the entities that only occur in the image and explore a more complete MNER task in the future.
Second, our work is a preliminary exploration of the GMNER task, and the proposed approaches are primarily based on previous representative NER or MNER methods. We hope this work can encourage more research to apply the recent advanced techniques from both the NLP and computer vision communities to improve its performance.
Ethics Statement
Our dataset is constructed based on two public MNER datasets, i.e., Twitter-15 (Zhang et al., 2018) and Twitter-17 (Yu et al., 2020). Three graduate students are employed as our annotators. The average time to annotate every 1,000 samples for each annotator is around 17 hours. Since the two datasets publicly released the text, images, and named entities, each annotator is asked to independently annotate the bounding box groundings for each entity without accessing to the user account. To ensure that the annotators were fairly compensated, we paid them at an hourly rate of CNY 36 (i.e., USD 5.2 per hour), which is higher than the current average wage in Jiangsu Province, China. We do not share personal information and do not release sensitive content that can be harmful to any individual or community. Because it is easy to retrieve multimodal tweets via image IDs from the two MNER datasets, we will release our annotation based on the textual modality and unique image IDs.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their insightful comments. This work was supported by the Natural Science Foundation of China (62076133 and 62006117), and the Natural Science Foundation of Jiangsu Province for Young Scholars (BK20200463) and Distinguished Young Scholars (BK20200018).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We report our limitations in the last section.
7 A2. Did you discuss any potential risks of your work? We do not think there are any potential risks of our work.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: We summarize our contributions in the introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We introduce our new dataset in section 3.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: The artifacts we use are referenced and briefly introduced in section 5.1.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use the publicly available Twitter datasets released by previous MNER works to study our GMNER task with further annotations. We will state the original licenses when releasing the dataset.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discuss the intended use of our proposed dataset in section 1.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data we use is based on the publicly available datasets, which have been checked and preprocessed by previous works.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We analyze our proposed Twitter-GMNER dataset in section3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We analyze our proposed Twitter-GMNER dataset in section3.
C 3 Did you run computational experiments? We introduce our experiments in section 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We introduce the experimental details in appendix A.3.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We introduce the experiment settings in section 5.1.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Because we have divided our dataset into train, dev, and test sets, we choose a model which obtains the best result on the dev set with a single run, and report its performance on the test set.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We introduce the parameter settings in section 5.1.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? We introduce human annotation details in section3 and appendix A.1.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? We introduce the annotation procedure in appendix A.1.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We introduce this information in appendix A.1.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We build the Twitter-GMNER dataset based on the public datasets and follow their usage requirements.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? The dataset we use is based on publicly available datasets, which have been approved by an ethics review board in previous works.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: We include this information in section 3.

heading: Future Work
text: Therefore, it would be interesting to further annotate the entities that only occur in the image and explore a more complete MNER task in the future. Second, our work is a preliminary exploration of the GMNER task, and the proposed approaches are primarily based on previous representative NER or MNER methods. We hope this work can encourage more research to apply the recent advanced techniques from both the NLP and computer vision communities to improve its performance