LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition (2024)

Abstract: Grounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal task that aims to identify named entities, entity types and their corresponding visual regions. GMNER task exhibits two challenging properties: 1) The weak correlation between image-text pairs in social media results in a significant portion of named entities being ungroundable. 2) There exists a distinction between coarsegrained referring expressions commonly used in similar tasks (e.g., phrase localization, referring expression comprehension) and finegrained named entities. In this paper, we propose RiVEG, a unified framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models (LLMs) as a connecting bridge. This reformulation brings two benefits: 1) It maintains the optimal MNER performance and eliminates the need for employing object detection methods to pre-extract regional features, thereby naturally addressing two major limitations of existing GMNER methods. 2) The introduction of entity expansion expression and Visual Entailment (VE) module unifies Visual Grounding (VG) and Entity Grounding (EG). It enables RiVEG to effortlessly inherit the Visual Entailment and Visual Grounding capabilities of any current or prospective multimodal pretraining models. Extensive experiments demonstrate that RiVEG outperforms state-of-the-art methods on the existing GMNER dataset and achieves absolute leads of 10.65%, 6.21%, and 8.83% in all three subtasks. Code and datasets publicly available at https: //github.com/JinYuanLi0012/RiVEG.

heading: 1 Introduction
text: Multimodal Named Entity Recognition (MNER) (Lu et al., 2018a) aims to extract named entities and corresponding categories from image-text pairs sourced from social media. As one of the most
* corresponding authors.
important subtask in multimodal information processing, MNER has received extensive attention in recent years (Wang et al., 2022c,b; Li et al., 2023; Cui et al., 2024). Due to the pressing demand for constructing structured knowledge graphs, humanmachine interaction and robotics, recent research endeavors to extend the scope of MNER (Wang et al., 2023). As a further extension, Grounded Multimodal Named Entity Recognition (GMNER) (Yu et al., 2023) aims to extract text named entities, entity types and their bounding box groundings from image-text pairs.
Analyzing GMNER from the two perspectives of MNER and EG, existing GMNER methods (Yu et al., 2023) have several significant limitations. Limitation 1. Entity Grounding (EG) is an important part in GMNER, which aims to determine the groundability and visual grounding results of text named entities predicted by the MNER model. Obviously, effective EG depends on accurate MNER. But the MNER performance of the existing multimodal fusion GMNER methods (Yu et al., 2023) is suboptimal. This may be due to the introduction of visual object features destroying the original
1302
text feature representation in MNER (Wang et al., 2022c). A series of MNER studies in the Text-Text paradigm (i.e., represent images using text) (Wang et al., 2022b; Li et al., 2023) show that text representation plays the most critical role in MNER. Models with the best MNER performance typically do not involve any cross-modal interactions. However, existing end-to-end GMNER methods (Yu et al., 2023) violate this in order to obtain EG capabilities with the help of cross-modal interactions. This results in the feature representation learned by the model being neither the best for MNER nor the best for EG. Therefore, splitting the prediction and grounding of named entities into separate stages is necessary. Limitation 2. Existing GMNER methods (Yu et al., 2023; Wang et al., 2023) typically employ object detection (OD) techniques to extract features from candidate regions, which are then aligned with text entities. But as shown in Table 1, our statistical results1 indicate that the widely-adopted OD methods exhibit unsatisfactory performance in recognizing candidate visual objects in GMNER dataset. This implies that there exists a natural upper limit to the effectiveness of existing GMNER methods. In instances where the OD methods are unable to detect the visual groundtruth bounding boxes for textual named entities, even if the MNER part predicts the correct textual named entity, the EG result will definitely be wrong. Therefore, it is necessary to avoid extracting visual features from candidate image regions.
Visual Grounding (VG) (Rohrbach et al., 2016; Yu et al., 2018) aims to correlate natural language descriptions with specific regions in images. Although VG has a similar paradigm to the Entity Grounding (EG) in GMNER, existing VG methods (Zhu et al., 2022; Deng et al., 2021; Wang et al., 2022a) exhibit significant deficiencies when applied to the GMNER dataset. This is attributed to two fundamental distinctions between these two tasks: Differentiation 1. Referring to the Pipeline-1 and Pipeline-2 in Figure 1, the text input for the EG task consists of real-world named entities, which is notably different from the referring expressions or noun phrases typically used in the VG method. There is a substantial difference that is almost nongeneralizable between them. The VG method can
1Here, TopN-Prec@0.5 refers to the accuracy metric in object detection methods across all samples. It measures the proportion of named entities where at least one of the Top-N predicted bounding boxes based on detection probability has an IoU of 0.5 or greater with the ground-truth bounding box.
comprehend the description “A female superhero in the MCU” but struggles with understanding named entities like “Black Widow”. Therefore, a bridge between noun phrases and named entities needs to be built. Differentiation 2. Due to the social media usage patterns of users, image-text pairs from social media exhibit apparent weak correlation. It implies that the textual named entity may not necessarily be linked to a specific region in the image. Yu et al. (2023) shows that approximately 60% of the named entities in the GMNER dataset are ungroundable. But the classic VG task stipulates that the input referring expression must match an object in the image, without considering ungroundable expressions that do not match any object. It means that the behavior of the existing VG methods is undefined if the named entity does not correspond to any object in the image. As shown in Pipeline-1 and Pipeline-2 of Figure 1, existing VG methods inevitably make errors when they confront the ungroundable text input like “Marvel Studios”. Our experiments demonstrate that the Prec@0.5 of the state-of-the-art VG methods applied to the GMNER dataset is 21.87% (Wang et al., 2022a) and 9.23% (Zhu et al., 2022), respectively.2 Therefore, a separate module is needed to determine the groundability of text inputs.
Given the above differences, the potential of VG methods cannot be unleashed in EG. Considering that most VG methods do not require preextraction of regional features, we ask: Is it possible to reformulate GMNER as a two-stage union of MNER-EG? The first stage focuses on optimizing the MNER effect to solve the Limitation 1 of the existing GMNER method. The second stage uses the VG method to naturally bypass regional feature extraction to solve Limitation 2. In the second stage, to adapt the VG method to the EG task, is it
2We fine-tune VG models using the GMNER dataset. Text inputs are named entities and entity types. For ungroundable entities, the coordinates of gold labels are defined as all zeros.
conceivable to establish a bridge between named entities and noun phrases to address Differentiation 1 between the VG method and EG task? Moreover, is it feasible to resolve Differentiation 2 by devising a sensible filtering mechanism?
In this paper, we introduce GMNER into a multistage framework consisting of MNER, VE and VG (RiVEG). This modular design inherently mitigates two limitations of existing GMNER methods, enabling the connection of three unrelated independent tasks in series within GMNER and unlocking significant potential. Specifically, in order to ensure the optimal performance of the MNER stage, we leverage auxiliary refined knowledge distilled from large language models (LLMs) to aid in the identification of named entities. And we introduce a Visual Entailment (VE) module to handle the weak correlation between image and text. Named entities are heuristically converted by LLMs into corresponding entity expansion expressions to facilitate the VE and VG modules to determine the groundability and grounding results of named entities. Main contributions are summarized as follows:
(i) RiVEG extends the text input of Visual Entailment and Visual Grounding from limited natural language expressions that require manual definition to infinite named entities that exist naturally by leveraging LLMs as bridges.
(ii) RiVEG increases the application scope of VG models to the cases of no right objects in image and demonstrates a new paradigm for GMNER. This paradigm supports data augmentation and facilitates subsequent upgrades.
(iii) All 14 variants of RiVEG achieve new state-of-the-art performance on the GMNER dataset. And after effective data augmentation, the MNER module of RiVEG achieves new state-of-the-art performance on two classic MNER datasets.

heading: 2 Related Work

heading: 2.1 Multimodal Named Entity Recognition
text: MNER is a relatively independent multimodal subtask. Early MNER methods attempt to simply interact text and images (Moon et al., 2018; Lu et al., 2018b; Yu et al., 2020). To deal with the limitations of this shallow interaction, subsequent methods explore various cross-modal fusion approaches (Zhang et al., 2021a; Wang et al., 2022e; Jia et al., 2023; Chen et al., 2022a,b). There is a trend
to solve MNER through the Text-Text paradigm (Wang et al., 2022c,b; Li et al., 2023), and this paradigm generally yields better performance. Despite obtaining promising results, these methods lack any visual understanding capabilities. The GMNER task poses significant challenges to the above research.

heading: 2.2 Grounded Multimodal Named Entity Recognition and Visual Grounding
text: GMNER is a further exploration of MNER. Existing GMNER works (Yu et al., 2023; Wang et al., 2023) aim to utilize OD methods for obtaining region proposals and subsequently sorting them based on their relevance to named entities. Such methods are inevitably limited by OD methods. Visual Grounding (VG) is a representative multimodal subtask. The progress in multimodal pretraining research (Zhu et al., 2022; Wang et al., 2022a; Yan et al., 2023) continues to contribute to the improvement of VG methods. However, the VG task has some strong predefined constraints. It presupposes that there are objects in the image that match the input text query. This renders VG methods inapplicable to the GMNER task that deviates from such predefined constraints. In this paper, we introduce the VE module to reconcile this conflict. Our method uses the VG method as the main body of visual understanding and achieves better performance than the existing GMNER methods.

heading: 3 Methodology
text: RiVEG is mainly divided into two stages. In the stage of Named Entity Recognition and Expansion, we employ auxiliary refined knowledge obtained from LLMs to ensure optimal MNER performance. Furthermore, in order to adapt existing VG methods to EG task, LLMs is considered as a bridge between named entities and referring expressions. RiVEG instructs LLMs to convert named entities into named entity referring expressions (detailed in §3.2). In the stage of Named Entity Grounding, RiVEG reformulates the entire EG task as a union of VE and VG. To reconcile the challenge between the weak correlation of image-text pairs in GMNER and the strong predefined constraints of existing VG methods, we design the VE module for buffering. Images and named entity referring expressions serve as the input to VE module for determining the groundability of the named entities. Finally, the named entities deemed groundable
alongside images are employed as inputs to VG module to obtain the final grounding results (detailed in §3.3). We also design a data augmentation method suitable for RiVEG. An overview of the RiVEG is depicted in Figure 2.

heading: 3.1 Task Formulation
text: Given a sentence S = {s1, · · · , sn} with n tokens and its corresponding image I , the objective of GMNER is to identify and categorize named entities (i.e., PER, LOC, ORG and MISC) within the sentence while concurrently determining the visually grounded region corresponding to each entity. Formulate that the output of GMNER comprises a set of multimodal entity triples:
Y = {(e1, t1, v1), · · · , (en, tn, vn)}
where ei is one of the entities in sentence, ti refers to the type of ei, and vi represents the corresponding visually grounded region. Note that if ei is ungroundable, vi is specified as None; otherwise, vi consists of a 4D coordinates representing the top-left and bottom-right locations of the grounded bounding box, i.e., (vx1i , v y1 i , v x2 i , v y2 i ).

heading: 3.2 Stage-1. Named Entity Recognition and Expansion

heading: Multimodal Named Entity Recognition Module
text: Incorporating document-level extended knowledge into original multimodal samples has been proven
to be one of the most effective methods for MNER (Wang et al., 2022c,b; Li et al., 2023). PGIM (Li et al., 2023) points out that despite the inherent limitations of generative models in sequence labeling tasks, LLM serve as an implicit knowledge base that is highly suitable for MNER task. We use the auxiliary refined knowledge3 provided by PGIM to enhance the predictive performance of named entities. And we additionally use the same algorithm to generate different auxiliary refined knowledge for the same sample from more versions of LLMs to perform data augmentation. Specifically, definite auxiliary refined knowledge retrieved by LLMs as Z = {z1, · · · , zm}. We take the concatenation of the original sentence S = {s1, · · · , sn} together with the auxiliary refined knowledge Z as the input of the transformer-based encoder:
embed([S;Z]) = {r1, · · · , rn, · · · , rn+m} And the acquired token representations R = {r1, · · · , rn} is fed into a linear-chain Conditional Random Field (CRF) (Lafferty et al., 2001) to generate the predicted sequence y = {y1, · · · , yn}:
P (y|S,Z) =
n∏ i=1 ψ(yi−1, yi, ri)
∑ y′∈Y n∏ i=1 ψ(y′i−1, y ′ i, ri)
3Original text and image captions are used as inputs to LLMs, and LLMs are guided to generate extended explanations of the original samples.
where ψ is the potential function and Y is the set of all possible label sequences given the input S and Z. Finally, given the input sequence with gold labels y∗, we employ the negative log-likelihood (NLL) loss function for training model:
LMNER(θ) = − logPθ(y∗|S,Z)
Note that the approach employed by this module is not unique and can be replaced or upgraded by any more powerful MNER methods in the future.
Entity Expansion Expression After extracting named entities in the sentence, RiVEG further guides LLMs to reformulate the fine-grained named entities into coarse-grained entity expansion expressions. Specifically, for each extracted named entity, we formulate it as the following template:
Background: {Image Caption} Text: {Sentence} Question: In the context of the provided information, tell me briefly what is the {Named Entity} in the Text? Answer: {Entity Expansion Expression}
where {Entity Expansion Expression} is reserved for LLMs to generate. A limited number of predefined in-context examples are employed to guide the format of responses (detailed in Appendix Figure 5). Finally, we concatenate named entities, entity categories, and entity expansion expressions into the final named entity referring expressions: Named Entity(Entity Categoriy) − Entity

heading: Expansion Expression
text: Furthermore, in order to perform data augmentation, we use different LLMs to generate different entity expansion expressions of the same entity.

heading: 3.3 Stage-2. Named Entity Grounding
text: Through the aforementioned transformation, the GMNER task is naturally reframed as the MNERVE-VG task. Notably, a large number of wellestablished studies exist on MNER, VE, and VG. The VE and VG methods that can be utilized at this stage are also not unique.
Visual Entailment Module For existing VG methods, fine-tuning with limited data hardly equips them with the ability to effectively handle irrelevant text and images. Hence, addressing the weak correlation between images and text in GMNER poses the primary challenge in integrating VG methods into GMNER. To achieve this objective, RiVEG introduces the VE module before the VG module.
For the classic VE task, denote the VE dataset DVE as:
DVE = {(i1, h1, l1), · · · , (in, hn, ln)}
where ii, hi, li represent an image premise, a text hypothesis and a class label, respectively. Three labels e, n or c are determined by the relationship conveyed by (ii, hi). Specifically, e (entailment) represents ii ⊨ hi, n (neutral) represents ii ⊭ hi ∧ ii ⊭ ¬hi, c (contradiction) represents ii ⊨ ¬hi. Similar but distinct, RiVEG defines the GMNER dataset DGMNER(VE) as:
DGMNER(VE) = {(i1, h1, l1), · · · , (im, hm, lm)}
where ii, hi, li represent an image that corresponds to a named entity, a named entity referring expression and a class label, respectively. Note that since the groundability of a named entity in GMNER dataset is unambiguous, only two labels e and c are retained. e represents hi is groundable and c represents hi is ungroundable. Finally, RiVEG replaces the original VE dataset with DGMNER(VE) to finetune the existing vanilla VE model. In the inference phase, only the samples deemed groundable by the fine-tuned vanilla VE model will be fed into the final VG module. For the remaining samples that are judged to be ungroundable, their entity grounding results are directly defined as None.
Visual Grounding Module Expansion expressions bring named entities closer to the referring expressions required by existing VG methods. And VE module effectively filters out ungroundable samples. Once there is no requirement for VG methods to assess the groundability of the text input, existing vanilla VG methods can be seamlessly applied to determine the visually grounded region of the named entity. Denote the subset of all groundable named entity samples in GMNER dataset as:
DGMNER(VG) = {(i1, h1, v1), · · · , (ik, hk, vk)}
where ii, hi, vi represent an image that corresponds to a named entity, a named entity referring expression and a visually grounded region of hi, respectively. Note that if a named entity in DGMNER(VG) corresponds to multiple different bounding boxes, we specify that the bounding box with the largest area is chosen as the unique gold label. Subsequently, RiVEG employs DGMNER(VG) to substitute the original VG dataset and fine-tune the existing vanilla VG model.

heading: 4 Experiments

heading: 4.1 Settings
text: Datasets Based on two benchmark MNER datasets, i.e., Twitter-2015 (Zhang et al., 2018) and Twitter-2017 (Yu et al., 2020), Yu et al. (2023) builds the Twitter-GMNER dataset by filtering samples with missing images or more than 3 entities belonging to the same type. More details about the training data for different modules and the TwitterGMNER dataset are provided in Appendix A.2.

heading: 4.2 Main Results
text: We compare RiVEG with all existing baseline methods in Table 2. Following Yu et al. (2023), we also report results on two subtasks of GMNER, including MNER and Entity Extraction & Grounding (EEG). MNER aims to identify entity-type pairs, while EEG aims to extract entity-region pairs. The first group of text-only methods focuses on extracting entity-type pairs from text inputs. And the region prediction is then specified as the majority class, i.e., None. The second group of multimodal methods comprises the EVG series baseline methods and the end-to-end H-Index method proposed by Yu et al. (2023). The experimental results demonstrate the superiority of RiVEG over previous methods. Compared with the previous state-of-the-art method H-Index, RiVEG exhibits significant advantages, achieving absolute improvements of 9.47%, 4.78% and 7.61% in all three tasks without any data augmentation. Additionally, data augmentation methods exhibit significant effects. After data augmentation, RiVEG achieves an additional 1.18%, 1.43%, and 1.22% performance improvement, respectively. This illustrates the rationality and effectiveness of our motivation to maximize the MNER effect and introduce the VE module and entity expansion expressions to use advanced VG methods to solve the GMNER task. We attribute the performance improvement of RiVEG to the following factors: 1) Maximizing the effectiveness of MNER significantly mitigates the issue of error propagation in GMNER predictions. Because if there is a prediction error in the entity or entity type within an entity-type-region prediction triplet, the triplet must be judged as an error. RiVEG alleviates this problem by making the MNER stage independent and introducing auxiliary knowledge. 2) RiVEG does not utilize the OD method for pre-extracting features from candidate regions. As shown in Table

heading: MNER Twitter-2015 Twitter-2017
text: 1, existing OD methods exhibit subpar performance on the Twitter-GMNER dataset. RiVEG leverages the feature of VG methods, eliminating the need for pre-extracting features and naturally addressing this deficiency. 3) RiVEG designs an independent VE module to specifically tackle the weak correlation between image-text pairs in GMNER task. This method of enabling the model to undergo targeted training based on data characteristics enables RiVEG to gain a better ability to determine the groundability of named entities.
Note that this result does not represent the performance upper limit of RiVEG. The modular design facilitates straightforward upgrades, selecting more advanced methods for different modules in the future is expected to yield improved performance.4

heading: 4.3 Detailed Analysis
text: Contributions of Different LLMs to Various Subtasks Table 3 shows the test set results after using the same MNER model trained on different LLM versions of the same dataset. The method of obtaining external knowledge is exactly the same as PGIM (Li et al., 2023). Compared with ITA (Wang et al., 2022c) (Using optical character recognition, object detection and image caption) and MoRe (Wang et al., 2022b) (Using Wikipedia), the auxiliary knowledge provided by LLMs is more helpful to MNER. The quality of knowledge pro-
4Furthermore, we test 10 other text and visual variations of RiVEG in detail in Appendix A.5 and A.6. Experimental results show that the performance of all 14 RiVEG variants is superior to the previous state-of-the-art method.

heading: LLMs in GMNERVE(Acc.) ALBEF OFAlarge(VE)

heading: LLMs in GMNERVG(Prec@0.5) SeqTR OFAlarge(VG)
text: vided by LLMs with a small number of parameters is usually inferior to ChatGPT used by PGIM. RiVEG performs better than past state-of-the-art methods.5 The training set of RiVEG includes responses from all five LLMs. And the dev set and test set are the same as PGIM. This indicates that utilizing diverse styles of external knowledge based on the same original sample enhances the generalization ability of model.
The named entities and their visual grounding results in the Twitter-GMNER dataset are deterministic, so using different LLMs to generate different named entity expansion expressions can build different LLM versions of the dataset. Table 4 shows the test set results of the same VE and VG models after training on different versions of the datasets. The training set of the Mix version is a direct merger of all five training sets, and the dev set is identical to the ChatGPT version. Using all versions of the named entity referring expression as text input performs significantly better than using raw named entities. LLMs with a large number of parameters generally generate better quality results than LLMs with a small number of parameters. The results of the Mix version show that more different styles of generated data can improve model performance. This is reflected more intuitively in Appendix A.6.
5This is average result from three different random seeds.

heading: VE Module Text Input (Acc.) ALBEF OFAlarge(VE)

heading: VG Module Text Input (Prec@0.5) SeqTR OFAlarge(VG)

heading: Named Entity Referring Expressions Analysis
text: We further investigate the performance of various methods in VE and VG modules when confronted with diverse text inputs. The named entity referring expression in DGMNER(VE/VG) is segmented into the named entity along with entity category and the entity expansion expression. Table 5 shows the test set performance of different methods after finetuning on DGMNER(VE/VG). In order to control variables, this experiment is conducted on the expansion expression generated by ChatGPT.
The experimental results demonstrate that when the text inputs are the complete named entity referring expressions, all methods achieve optimal results. When the text inputs are named entities or entity expansion expressions, the performance of all methods decreases to a certain extent. Moreover, the sensitivity of different methods to various text inputs varies significantly. Unified frameworks like OFA demonstrate significantly better performance on fine-grained named entities than on coarse-grained entity expansion expressions. Contrarily, specialized methods like SeqTR, designed for a specific task, exhibit the opposite behavior. This experiment verifies the rationality of our proposed named entity referring expression. While different methods exhibit distinct characteristics, the expression method designed by us that combines both coarse and fine granularity emerges as their optimal choice.
Effect Analysis of Additional Training Data As the DGMNER(VE/VG) datasets we create differ only in text type from the SNLI-VE (Xie et al., 2019) and RefCOCO series datasets (Mao et al., 2016; Yu et al., 2016; Nagaraja et al., 2016), we investigate whether pretraining with more similar task data could yield additional performance improvements in Table 6.
For various VE methods, incorporating additional fine-tuning on the SNLI-VE dataset (Xie et al., 2019) prior to the final fine-tuning on the DGMNER(VE) does not result in improved performance. This observation suggests a significant difference between the DGMNER(VE) and the SNLIVE. For various VG methods, conducting additional fine-tuning on the RefCOCO series of datasets (Mao et al., 2016; Yu et al., 2016; Nagaraja et al., 2016) prior to the final fine-tuning on the DGMNER(VG) may result in enhanced performance. This may be attributed to a certain degree of similarity between the two. Because we convert named entities into named entity referring expressions through entity expansion expressions.6

heading: 5 Conclusion
text: In this paper, we propose RiVEG, a unified framework that aims to use LLMs as bridges to reformulate the GMNER task into a unified MNER-VE-VG task. This reformulation enables the framework to seamlessly incorporate the capabilities of stateof-the-art methods for each subtask in their corresponding modules. Extensive experiments show that RiVEG significantly outperforms state-of-theart methods. The proposed RiVEG greatly reduces the constraints of VG and VE methods on natural language inputs, expands their applicability to more realistic scenarios, and provides insights for future work to better deploy VG and VE modules in related tasks.
6ALBEF (Li et al., 2021) authors do not provide pretrained weights based on the SNLI-VE dataset. And some of the pretrained weights of SeqTR (Zhu et al., 2022) based on the RefCOCO series dataset cannot be obtained. Therefore, we follow the wishes of author and only test OFA(Wang et al., 2022a) in this experiment.

heading: Limitations
text: In the past, the three-stage reformulation of GMNER was not promising by researchers. Because there may be two intuitive limitations to this reformulation. Limitation 1. Splitting the entire pipeline into three stages can lead to severe error propagation. Limitation 2. Pipeline design seems to be quite heavy. But actually, regarding Limitation 1, this paper demonstrates for the first time that the three-stage pipeline design performs significantly better than existing end-to-end methods. Regarding Limitation 2, the modular nature of RiVEG allows it to freely select lighter sub-models to form more efficient variants to deal with time-critical scenarios. The most lightweight variant of RiVEG can complete the entire three-stage training in 5 hours using a single 4090 GPU, making the training cost quite low. And the inference speed of RiVEG is also acceptable, since LLMs can be called in the form of API. Considering the substantial performance improvement, sacrificing a certain degree of inference speed is worthwhile. In the future, our research direction is to strive for optimal performance while designing each module in a lightweight style and training them in conjunction, possibly incorporating intermediate auxiliary losses.

heading: Ethics Statement
text: In this paper, we use publicly available TwitterGMNER dataset for experiments. For the auxiliary refined knowledge and the entity expansion expression, RiVEG generates them using ChatGPT, Vicuna, LlaMA2. Therefore, we trust that all the data we use does not violate the privacy of any user.

heading: Acknowledgments
text: The authors would like to thank four anonymous reviewers for their insightful comments. This work was supported by the Natural Science Foundation of Tianjin (No.21JCYBJC00640) and the 2023 CCF-Baidu Songguo Foundation (Research on Scene Text Recognition Based on PaddlePaddle).

heading: A Appendix

heading: A.1 Model Configuration
text: In order to reasonably evaluate the capabilities of RiVEG, we choose advanced methods for each of three sub-modules of RiVEG. Specifically, for MNER module, RiVEG uses the same architecture as PGIM (Li et al., 2023), and selects BERTbase (Devlin et al., 2019) and XLM-RoBERTalarge (Conneau et al., 2020) as the text encoder, aligning with the prevailing MNER methods (Wang et al., 2022b,d,e). mPLUG-Owl (Ye et al., 2023) is employed to generate the image caption necessary for the entity expansion expression. For VE module, on the basis of OFAlarge (Wang et al., 2022a), RiVEG replaces its original VE dataset SNLI-VE (Xie et al., 2019) with DGMNER(VE) and DGMNER(VE)† for fine-tuning. For VG module, RiVEG chooses OFAlarge (Wang et al., 2022a), which also possesses visual grounding capability, to substitute its original VG datasets RefCOCO (Yu et al., 2016), RefCOCO+ (Yu et al., 2016), and RefCOCOg (Mao et al., 2016; Nagaraja et al., 2016) with DGMNER(VG) and DGMNER(VG)† for finetuning.

heading: A.2 Details of Different Module Datasets
text: Table 7 counts the training data used by different modules. And Table 8 shows the details of the Twitter-GMNER dataset. For the MNER module of RiVEG, the training data for base version comprises original samples and corresponding auxiliary refined knowledge generated by gpt-3.5-turbo provided by PGIM (Li et al.,

heading: Split #Tweet #Entity #Groundable Entity #Box
text: 2023). Additionally, data augmentation is applied to the training set of base version. In data augmentation version, we apply four additional LLMs (i.e., vicuna-7b-v1.5, vicuna-13b-v1.5, llama-2-7b-chat-hf, llama-2-13b-chat-hf) to training set samples using the same algorithm, generating multiple versions of auxiliary refined knowledge for the same sample. The dev set and test set remain the same as base version to ensure the benchmarking of evaluation.
For VE module, the training data of base version DGMNER(VE) exclusively comprises entity expansion expressions generated by gpt-3.5-turbo. In the data augmentation version DGMNER(VE)† , we also apply four additional LLMs to generate additional training samples and maintain the dev set and test set identical to base version DGMNER(VE).
For VG module, the training data of base version DGMNER(VG) is a subset of all groundable entity expansion expressions in base version DGMNER(VE). Similarly, the data augmentation version DGMNER(VG)† is a subset of the data augmentation version DGMNER(VE)† .

heading: A.3 Detailed Implementation Details
text: For the methods used by three modules of RiVEG, we only replace their original VE and VG datasets with DGMNER(VE)/DGMNER(VE)† and DGMNER(VG)/DGMNER(VG)† and primarily adhere to original hyperparameter configurations and training strategies as reported in their paper for finetuning. Without specific instructions, all training is conducted using a single 4090 GPU. The model
with the best results on each dev set is selected to evaluate the performance on each test set and used to perform the final inference. Note that considering the notable effectiveness of RiVEG, we refrain from conducting attempts with excessive hyperparameter combinations. Thus, exploring more hyperparameter combinations may yield improved results.
MNER Module RiVEG mainly follows MNER fine-tuning strategy of PGIM7 (Li et al., 2023). For the XLM-RoBERTalarge version in Table 2, Table 9 and Table 10, learning rate is set to 7e-6 and batchsize is set to 4. For the BERTbase version in Table 10, learning rate is set to 5e-5 and batchsize is set to 16. Remaining settings remain unchanged. The model is fine-tuned for 25 epochs.
VE Module For OFAlarge(VE) (Wang et al., 2022a), the learning rate is set to 2e-5 and batchsize is set to 32. We convert the labels entailment/contradiction to yes/no. And we also use the Trie-based search strategy to constrain the generated labels over the candidate set. Remaining settings follow their VE fine-tuning strategy8. The basic version of OFAlarge is fine-tuned for 6 epochs on DGMNER(VE)/DGMNER(VE)† .
For ALBEF (Li et al., 2021), the learning rate is set to 2e-5 and batchsize is set to 24. Remaining settings follow their VE fine-tuning strategy9. We finetune for 5 epochs on DGMNER(VE)/DGMNER(VE)† using a single V100 32G GPU based on ALBEF14M.
VG Module For OFAlarge(VG) (Wang et al., 2022a), learning rate is 3e-5 with the label smoothing of 0.1 and batchsize is set to 32. The input image resolution is set to 512×512. And the basic version of OFAlarge is fine-tuned for 10 epochs
7https://github.com/JinYuanLi0012/PGIM 8https://github.com/OFA-Sys/OFA#
visual-entailment 9https://github.com/salesforce/ALBEF# visual-entailment
on DGMNER(VG)/DGMNER(VG)† . Remaining settings follow their VG fine-tuning strategy10.
For SeqTR (Zhu et al., 2022), since SeqTR does not provide pretrained weights, we select the weight which are pretrained and fine-tuned for 5 epochs on RefCOCO11 (Yu et al., 2016), as the initial weight and fine-tune for 5 epochs on DGMNER(VG)/DGMNER(VG)† . The input image resolution is set to 640×640. Text length is trimmed to 30. The learning rate is 5e-4 and batchsize is 64. Visual encoder is YOLOv3 (Redmon and Farhadi, 2018). Remaining settings follow their Referring Expression Comprehension fine-tuning strategy12.

heading: A.4 Evaluation Metrics
text: The GMNER prediction is composed of entity, type, and visual region. Following Yu et al. (2023), the correctness of each prediction is computed as follows:
Ce/Ct =
{ 1, pe/pt = ge/gt;
0, otherwise.
Cv =    1, pv = gv = None;
1, max(IoU1, · · · , IoUj) > 0.5; 0, otherwise.
correct = { 1, Ce ∧ Ct ∧ Cv = 1; 0, otherwise.
10https://github.com/OFA-Sys/OFA# visual-grounding-referring-expression-comprehension
11https://github.com/seanzhuh/SeqTR#refcoco 12https://github.com/seanzhuh/SeqTR#
pre-training--fine-tuning
where Ce, Ct and Cv represent the correctness of entity, type and region predictions; pe, pt and pv represent the predicted entity, type and region; ge, gt and gv represent the gold entity, type and region; and IoUj denotes the IoU score between pv with the j-th ground-truth bounding box gv,j . Then precision (Pre.), recall (Rec.) and F1 score are used to evaluate the performance of model:
Pre = #correct
#predict , Rec =
#correct
#gold
F1 = 2× Pre×Rec Pre+Rec
where #correct, #predict and #gold respectively represent the number of triples of correct predictions, predictions and gold labels.

heading: A.5 Exploration of More Visual Variants
text: To further demonstrate the effectiveness of RiVEG, we present the performance of various RiVEG variants across all three subtasks in Table 9. Specifically, we additionally choose an early classic multimodal pretraining method ALBEF-14M (Li et al., 2021) for VE module and an early classic VG method SeqTR (Zhu et al., 2022) for VG module. Different combinations of methods are considered as different variants. Experimental results show that, without considering any data augmentation, the weakest variant still exhibits highly competitive results with the previous state-of-the-art method. And all the variants after data augmentation are significantly better than the baseline.
We also present the corresponding test set results for different methods of different modules
after fine-tuning. The results indicate that stronger combinations of sub-methods generally yield superior performance. The performance of the early classic ALBEF and SeqTR is inferior to the subsequent advanced OFA. Additionally, while the test set Prec@0.5 of SeqTR is slightly lower than that of OFAlarge(VG), there are significant differences in their final GMNER and EEG results. This phenomenon may be attributed to the text representation method employed by SeqTR. The GRU (Chung et al., 2014) vocabulary of SeqTR is solely statistically obtained from DGMNER(VG) or DGMNER(VG)† , rendering it incapable of handling out-of-vocabulary words in new samples during inference.
Moreover, data augmentation methods exhibit varying effects on distinct modules. MNER and VG methods can intuitively benefit from data augmentation, but this is not the case for VE methods. One potential reason is that the distribution after fitting more training data does not align with the optimal distribution of the test data. But this is acceptable since data augmentation significantly enhances the performance of overall framework.

heading: A.6 Exploration of More Text Variants
text: Table 10 explores the influence of various text encoders on the framework. Specifically, we fix the VE and VG modules and replace the originally used XLM-RoBERTalarge in the MNER module
with BERTbase. The main conclusions are as follows: 1) The weaker text encoder evidently results in inferior MNER performance compared with the XLM-RoBERTa variant. Due to the error propagation characteristics of GMNER prediction triples, weak MNER performance naturally results in the degradation of GMNER and EEG performance. 2) Concerning the MNER results, the BERT version of RiVEG outperforms the BERT version of Unimodal Baseline, surpassing 4.96% without any data augmentation and achieving a further lead of 6.26% after data augmentation. It underscores the effectiveness of incorporating auxiliary knowledge into the original samples in MNER task.
Additionally, to control variables and compare entity grounding performance across different methods, we conduct more challenging tests for RiVEG. The variants in the w/o Auxiliary Refined Knowledge in MNER group indicate that the text input to their MNER module consists only of the original text and does not incorporate any external knowledge. In this scenario, the MNER performance of these BERT variants of RiVEG is nearly identical to that of the two baseline methods. However, the GMNER and EEG performance of these variants still significantly outperforms all baseline methods. This comparison highlights the robust performance of RiVEG in entity grounding.
The last four lines of experimental results also illustrate the efficacy of the data augmentation
method. When facing the same named entity recognition and expansion results, VE and VG models trained using data augmentation methods generally achieve better GMNER and EEG results. This echoes the experimental results in Table 4.

heading: A.7 Case Study
text: We further conduct case study to compare the predictions of H-Index (Yu et al., 2023) and RiVEG for challenging samples. Figure 3 illustrates two examples where H-Index makes errors, while RiVEG produces accurate predictions. For the test sample on the left, even though H-Index correctly predicts the entity-type pairs, it fails to achieve any effective entity grounding. All predictions made by RiVEG are accurate, even for the challenging entity grounding case of the “Preds” team logo. For the sample on the right, H-Index fails to predict all named entities and does not perform entity grounding correctly. RiVEG completes predictions perfectly, even though the grounded area of “Nike SNKRS” is very small.
Figure 4 illustrates two error examples. The left test sample poses a significant challenge for entity grounding due to the resemblance between the shape of the clothing in the image and that of humans. H-Index fails to predict the named entity. While RiVEG accurately predicts the entity-type pairs, it incorrectly performs entity grounding. The entity prediction and grounding for the sample on the right are both highly challenging. In this case, H-Index is almost incapable of making effective predictions. RiVEG accurately predicts all entitytype pairs and successfully performs the grounding of “taylor swift”. However, it fails to accurately complete the grounding of “the black eyed peas”. This indicates that the GMNER task remains highly challenging.

heading: Future Work
text: In the future, our research direction is to strive for optimal performance while designing each module in a lightweight style and training them in conjunction, possibly incorporating intermediate auxiliary losses