VP-MEL: Visual Prompts Guided Multimodal Entity Linking
Authors: Hongze Mi, Jinyuan Li, Zhangxuying, Haoran Cheng, Jiahao Wang, Di Sun, Gang Pan
Accepted at: acl25

VP-MEL: Visual Prompts Guided Multimodal Entity Linking

Hongze Mi1, Jinyuan Li2, Xuying Zhang1, Haoran Cheng1, Jiahao Wang1, Di Sun3, Gang Pan1,2,*
1College of Intelligence and Computing, Tianjin University
2School of New Media and Communication, Tianjin University
3College of Artificial Intelligence, Tianjin University of Science and Technology
{mhzqs, jinyuanli, wjhwt, pangang}@tju.edu.cn

Abstract

Multimodal entity linking (MEL), a task aimed at linking mentions within multimodal contexts to their corresponding entities in a knowledge base (KB), has attracted much attention due to its wide applications. However, existing MEL methods primarily rely on mention words as retrieval cues, which limits their ability to effectively utilize both textual and visual information. As a result, MEL struggles to retrieve entities accurately, particularly when the focus is on image objects or when mention words are absent from the text. To address these issues, we introduce Visual Prompts guided Multimodal Entity Linking (VP-MEL). Given a text-image pair, VP-MEL links a marked image region (i.e., visual prompt) to its corresponding KB entity. To support this task, we construct VPWiki, a dataset specifically designed for VP-MEL. Additionally, we propose the Implicit Information-Enhanced Reasoning (IIER) framework, which enhances visual feature extraction through visual prompts and leverages the pre-trained Detective-VLM model to capture latent information. Experimental results on VPWiki demonstrate that IIER outperforms baseline methods across multiple benchmarks for VP-MEL. Code and datasets will be released at https://github.com/MiHongze-tju/VP-MEL.

1 Introduction

Linking ambiguous mentions with multimodal contexts to the referent unambiguous entities in a knowledge base (KB), known as Multimodal Entity Linking (MEL) (Moon et al., 2018), is an essential task for various multimodal applications. Most MEL works (Gan et al., 2021; Wang et al., 2022a; Dongjie and Huang, 2022; Luo et al., 2023; Zhang et al., 2023a; Xing et al., 2023; Shi et al., 2024) mainly focus on improving the interaction of multimodal information and achieve promising performance. However, existing methods typically represent mentions in the form of mention words and assume that each mention is associated with a high-quality image, which results in two limitations for MEL.

Text information dependency: MEL primarily relies on mention words for entity linking, as these words frequently exhibit significant overlap with entity names in real-world applications. Such overlap serves as a strong cue for identifying entities within the knowledge base(KB). However, MEL performs poorly when mention words are absent or unannotated. As shown in Figure 1, without annotated mention words, MEL computes similarity based on the entire text, which can lead to erroneous entity linking. For instance, MEL may incorrectly associate the data with the entity Olympic Games due to high textual similarity. Since Lee C W and Lin Dan are not explicitly mentioned, MEL fails to establish correct links to these entities. This issue underscores the difficulty of MEL in accurately linking data when the mention words related to entities are missing from the data.

![Comparison between MEL and VP-MEL tasks. MEL is typically limited to selecting mentions from text. In contrast, VP-MEL addresses this limitation by using visual prompts to link specific regions in the images to the correct entities in the knowledge base.](page_765_370_496_312.png)

Figure 1: Comparison between MEL and VP-MEL tasks. MEL is typically limited to selecting mentions from text. In contrast, VP-MEL addresses this limitation by using visual prompts to link specific regions in the images to the correct entities in the knowledge base.
Image modality impurity: Compared to textual data, images often exhibit higher levels of noise. Misinterpreting or misusing image information can substantially impact MEL performance. Most existing coarse-grained methods (Yang et al., 2023; Li et al., 2024b; Luo et al., 2023; Wang et al., 2022a; Li et al., 2023a) directly encode the entire image, making it difficult to eliminate noise interference. Song et al. (2024) enhances MEL performance by extracting fine-grained image information through object detection. However, this approach still relies on sufficient textual information for accurate object localization and is prone to interference from visually similar objects. Therefore, a potentially effective strategy to mitigate image noise is enhancing object localization precision while reducing the dependence on textual data.

These limitations hinder the ability of MEL to fully exploit multimodal data effectively. Despite being fundamental to multimodal data, images contribute minimally to MEL. Furthermore, the strong reliance on textual data significantly limits MEL performance, especially when the text is scarce or incomplete. So we ask: Is it possible to link specific objects in multimodal images to the KB even with insufficient textual information? Investigating this possibility could unlock the full potential of multimodal data for MEL.

In this paper, we introduce Visual Prompts guided Multimodal Entity Linking (VP-MEL), a new task designed for entity linking in image-text pairs, as shown in Figure 1. VP-MEL annotates mentions directly on images using visual prompts, eliminating reliance on textual mention words. This approach broadens its applicability, enabling effective linking of multimodal data to the KB even when textual information is limited or image-based information is prioritized. To support this task, we develop the VPWiki dataset by extending existing public MEL datasets, where visual prompts are assigned to each mention within the corresponding images.

To tackle the challenges of VP-MEL, we propose the Implicit Information-Enhanced Reasoning (IIER) framework. IIER leverages visual prompts as guiding texture cues to focus on specific local image regions. To reduce reliance on textual data, it employs an external implicit knowledge base to heuristically generate auxiliary information for the reasoning process. Specifically, a CLIP visual encoder is employed to extract both global image features and local features guided by visual prompts. Additionally, a Vision-Language Model(VLM) incorporating a CLIP visual encoder is pre-trained to generate textual information from visual prompts, supplementing existing text data. IIER integrates both supplementary visual and textual information, enhancing the linking of objects in images to the KB.

Main contributions are summarized as follows:

(i) We introduce VP-MEL, a new entity linking task that replaces traditional mention words with visual prompts, linking specific objects in images to the KB.

(ii) We develop VPWiki, a high-quality annotated dataset, to establish a strong benchmark for evaluating VP-MEL. Furthermore, we introduce an automated annotation pipeline to improve the efficiency of VPWiki dataset construction.

(iii) We propose the IIER framework to tackle VP-MEL by effectively leveraging multimodal information and reducing reliance on a single modality. Compared to previous methods, IIER achieves a 20% performance improvement in the VP-MEL task and maintains competitive results in the MEL task.

2 Related Work

2.1 Multimodal Entity Linking

Given the widespread use of image-text content in social media, the integration of both modalities for entity linking is essential. For example, Moon et al. (2018) pioneer the use of images to aid entity linking. Building on this, Adjali et al. (2020) and Gan et al. (2021) construct MEL datasets from Twitter and long movie reviews. Expanding the scope of MEL datasets, Wang et al. (2022c) present a high-quality MEL dataset from Wikinews, featuring diversified contextual topics and entity types. To achieve better performance on these datasets, a multitude of outstanding works in the MEL field (Wang et al., 2022a; Yang et al., 2023; Luo et al., 2023; Shi et al., 2024) emerge, focusing on extracting and interacting with multimodal information. Song et al. (2024) use object detection to extract visual information from images and better link mention words to correct entities, but still face difficulty in linking images to KBs in the absence of mention words. Although multimodal information can enhance entity linking performance, in these methods, text consistently dominates over images.
2.2 Vision Prompt
Region-specific comprehension in complex visual scenes has become a key research topic in the field of Multimodal Computer Vision. Existing methods typically utilize textual coordinate representations (Zhu et al., 2024; Zhao et al., 2023), learned positional embeddings (Peng et al., 2024; Zhang et al., 2023b; Zhou et al., 2023), or Region of Interest (ROI) features (Zhang et al., 2023b) to anchor language to specific image regions. More recently, Cai et al. (2024) propose a coarse-grained visual prompting solution that directly overlays visual prompts onto the image canvas. In contrast, our VP-MEL provides a fine-grained entity linking method based on visual prompts to reduce reliance on text.

3 Dataset
As there is no existing MEL dataset that incorporates visual prompts, constructing a high-quality dataset is essential for establishing a strong benchmark for the VP-MEL task.

Data Collection. Our dataset is built based on two benchmark MEL datasets, i.e., WikiDiverse (Wang et al., 2022c) and WikiMEL (Wang et al., 2022a). Appendix A.7 provides detailed information.

Annotation Design. Given an image-text pair with corresponding mention words, annotators are required to: 1) identify and annotate relevant visual prompts in the image based on the mention words; 2) discard samples where the image and mention words are unrelated; 3) refine annotations for samples with inaccurate automated labels; 4) assign an entity type to each instance (i.e., Person, Organization, Location, Country, Event, Works, Misc.).

Annotation Procedure. To improve data annotation efficiency, we develop a pipeline that automatically annotates visual prompts in images based on mention words inspired by Li et al. (2024a). In the pipeline, the Visual Entailment Module is employed to evaluate and filter out the highly relevant data. Subsequently, the Visual Grounding Module annotates the visual prompts in the images. The details of the pipeline are provided in Appendix A.6. The annotation team consists of 10 annotators and 2 experienced experts. All annotators have linguistic knowledge and are instructed with detailed annotation principles. Fleiss Kappa score (Fleiss, 1971) of annotators is 0.83, indicating strong agreement among them. We employ the Intersection over Union (IoU) metric to assess annotation quality and discard samples with an IoU score below 0.5.

<table>
  <tr>
    <th></th>
    <th>Train</th>
    <th>Dev.</th>
    <th>Test</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>pairs</td>
    <td>8,000</td>
    <td>1,035</td>
    <td>1,052</td>
    <td>10,087</td>
  </tr>
  <tr>
    <td>ment. per pair</td>
    <td>1.18</td>
    <td>1.16</td>
    <td>1.27</td>
    <td>1.19</td>
  </tr>
  <tr>
    <td>words per pair</td>
    <td>9.89</td>
    <td>9.80</td>
    <td>10.32</td>
    <td>9.92</td>
  </tr>
</table>

Table 1: Statistics of VPWiki. ment. denotes Mentions.

![An example from VPWiki. GT denotes the ground truth entity. The red box in the left image represents the visual prompt annotated for the VP-MEL task.](page_1012_1012_370_180.png)

Figure 2: An example from VPWiki. GT denotes the ground truth entity. The red box in the left image represents the visual prompt annotated for the VP-MEL task.

![More statistics of VPWiki. (a) Distribution of entity types. (b) Distribution of the number of candidate entities per mention.](page_1012_1342_370_180.png)

Figure 3: More statistics of VPWiki. (a) Distribution of entity types. (b) Distribution of the number of candidate entities per mention.

Dataset Analysis. Figure 2 illustrates an example from the VPWiki dataset. Additional data samples are provided in A.9. The VPWiki dataset comprises a total of 12,720 samples, which are randomly split into training, validation, and test sets with an 8:1:1 ratio. Detailed statistics of the VPWiki dataset are provided in Table 1. Additionally, Figure 3 presents the distribution of entity types and the number of candidate entities per mention in the dataset. In Figure 3(a), abbreviations are used to represent each entity type. Meanwhile, Figure 3(b) shows that as the number of candidate entities per mention increases, the task becomes increasingly challenging.
4 Task Formulation

The multimodal knowledge base consists of a set of entities \( \mathcal{E} = \{ E_i \}_{i=1}^N \), where each entity is represented as \( E_i = (e_{n_i}, e_{v_i}, e_{d_i}, e_{a_i}) \). Here, \( e_{n_i} \) represents the entity name, \( e_{v_i} \) denotes the entity images, \( e_{d_i} \) corresponds to the entity description, and \( e_{a_i} \) encodes the entity attributes. A mention is denoted as \( M_j = (m_{s_j}, m_{v_j}) \), where \( m_{s_j} \) represents the sentence and \( m_{v_j} \) corresponds to the corresponding image. The corresponding entity of mention \( M_j \) in the knowledge base is denoted as \( E_i \). The objective of the VP-MEL task is to retrieve the ground truth entity \( E_i \) from the entity set \( \mathcal{E} \) in the knowledge base, based on \( M_j \).

5 Methodology

In this section, we describe the proposed IIER framework for the VP-MEL task. As illustrated in Figure 4, IIER utilizes visual encoder to extract both deep semantic features and shallow texture features, which are enhanced by visual prompts (\S 5.1). To avoid excessive reliance on visual features, the Detective-VLM module is designed to generate supplementary textual information guided by visual prompts(\S 5.2), which is then combined with the original text and processed by the text encoder (\S 5.3). Finally, a similarity score is computed after integrating the visual and textual features(\S 5.4).

5.1 Visual Encoder

We choose pre-trained CLIP model (Dosovitskiy et al., 2021) as our visual encoder. Extensive research (Cai et al., 2024; Shtedritski et al., 2023) demonstrates its effectiveness in interpreting visual markers. The image \( m_{v_j} \) of \( M_j \) is reshaped into n 2D patches. After this, image patches are processed through visual encoder to extract features. The hidden states extracted from \( m_{v_j} \) by the CLIP visual encoder are represented as \( V_{M_j}^l = [v_{[CLS]}^0; v_{M_j}^1; v_{M_j}^2; ...; v_{M_j}^n] \in \mathbb{R}^{(n+1) \times d_e} \), where \( d_e \) denotes the dimension of the hidden state and \( l \) denotes the number of layers in the encoder.

Since CLIP focuses on aligning deep features between images and text and may overlook some low-level visual details (Zhou et al., 2022), we selectively extract features from both the deep and shallow layers of CLIP. Specifically, a shallow feature (\( V_{M_j}^3 \)) is used to represent the textures and geometric shapes in the image, while deep features (\( V_{M_j}^{10}, V_{M_j}^{11}, V_{M_j}^{12} \)) are used to represent abstract semantic information. We take the hidden states corresponding to the special [CLS] token (\( v_{[CLS]}^0 \in \mathbb{R}^{d_e} \)) from these layers as the respective visual features \( F^l \). These features are concatenated and normalized using LayerNorm, and then passed through a MLP layer to transform the dimensions to \( d_v \), with the output representing the global features of the image \( V_{M_j}^G \in \mathbb{R}^{d_v} \).

\[
F^l = v_{[CLS]}^0 \in V_{M_j}^l,
\]
\[
V_{M_j}^{G'} = \mathrm{LN}\left( \mathrm{Concat}(F^3, F^{10}, F^{11}, F^{12}) \right),
\]
\[
V_{M_j}^G = \mathrm{MLP}\left( V_{M_j}^{G'} \right).
\]

Then, hidden states from the output layer of encoder \( V_{M_j}^l \) are passed through a fully connected layer, which also transforms the dimensions to \( d_v \), yielding the local features of the image \( V_{M_j}^L \in \mathbb{R}^{(n+1) \times d_v} \):

\[
V_{M_j}^L = \mathrm{FC}\left( V_{M_j}^l \right).
\]

For the image \( e_{v_i} \) of entity \( E_i \), the global feature \( V_{E_i}^G \) and local feature \( V_{E_i}^L \) are obtained using the same method described above.

5.2 Detective-VLM

Real-world multimodal data often contain challenges such as short texts or image noise. In this context, VLMs serve as implicit knowledge bases, can analyze both image and text to infer useful auxiliary information. Most VLMs (Liu et al., 2024; Zhu et al., 2024; Ye et al., 2023; Li et al., 2023b) adopt the CLIP visual encoder, enabling them to focus more effectively on markers in images compared to other visual methods (Cai et al., 2024; Shtedritski et al., 2023). Therefore, we instruction fine-tune a VLM to extract effective information from images. The VLM follows template designed below to further mine potential information from the image \( m_{v_j} \) and sentence \( m_{s_j} \) of mention \( M_j \), assisting in subsequent feature extraction:

<table>
  <tr>
    <th>Background:</th>
    <td>{Image}</td>
  </tr>
  <tr>
    <th>Text:</th>
    <td>{Sentence}</td>
  </tr>
  <tr>
    <th>Question:</th>
    <td>Based on the text '{Sentence}', tell me briefly what is the {Entity Type} and {Entity Name} in the red box of the {Image}?</td>
  </tr>
  <tr>
    <th>Answer:</th>
    <td>{Entity Name} {Entity Type}</td>
  </tr>
</table>

We utilize VPWiki dataset to design the fine-tuning dataset, where {Image} and {Sentence} correspond to \( m_{v_j} \) and \( m_{s_j} \) in \( M_j \), respectively. During the inference process, {Entity Name}
Figure 4: The overall architecture of Implicit Information-Enhanced Reasoning (IIER) framework. The image-text pairs of the Mention and Entity are used together as input. Specifically, Mention Text is the sentence corresponding to Mention Image, while Entity Text consists of Entity Name and Entity Attribute corresponding to the Entity Image in the Knowledge Base.

and \{Entity Type\} need to be generated by VLM. Details of the dataset and Detective-VLM can be found in Appendix A.4.

The objective formula for instruction fine-tuning Detective-VLM is expressed as follows:

\[
\min_{\theta} \sum_{i=1}^{N} \mathcal{L}(f_{\theta}(x_i), y_i),
\]

where \( f \) represents the pre-trained VLM, and \( \theta \) denotes the model parameters. \( N \) represents the number of instruction-output pairs, \( x_i \) is the \( i \)-th instruction, and \( y_i \) is the corresponding desired output. \( \mathcal{L} \) is defined as:

\[
\mathcal{L}(f_{\theta}(x_i), y_i) = - \sum_{t=1}^{T} \log P_{\theta}(y_i^{(t)}|x_i),
\]

where \( T \) is the length of output sequence, \( y_i(t) \) is the \( t \)-th word of the expected output \( y_i \) at time step \( t \), and \( P \) is conditional probability that the model generates the output \( y_i(t) \) at time step \( t \).

Detective-VLM aims to ensure that the output is both accurate and relevant, minimizing the likelihood of generating irrelevant information. Notably, we represent the Answer output by VLM as \( m_{w_j} \).

5.3 Textual Encoder

For the mention \( M_j \), after concatenating mention sentence \( m_{s_j} \) with \( m_{w_j} \), they form the input sequence, with different parts separated by [CLS] and [SEP] tokens:

\[
I_{M_j} = [CLS] \ m_{w_j} \ [SEP] \ m_{s_j} \ [SEP].
\]

Hidden states of output layer after the input sequence passes through text encoder are represented as \( T_{M_j} = [t_{[CLS]}^0; t_{M_j}^1; ...; t_{M_j}^{l_t}] \in \mathbb{R}^{(l_t+1) \times d_t} \), where \( d_t \) represents the dimension of output layer features, and \( l_t \) denotes the length of input. We use the hidden state corresponding to [CLS] as the global feature of the text \( T_{M_j}^G \in \mathbb{R}^{d_t} \), and the entire hidden states as the local features of the text \( T_{M_j}^{L_t} \in \mathbb{R}^{(l_t+1) \times d_t} \).

The input sequence for entity \( E_i \) consists of the entity name \( e_{n_i} \) and entity attributes \( e_{a_i} \), which can be represented as:

\[
I_{E_i} = [CLS] \ e_{n_i} \ [SEP] \ e_{a_i} \ [SEP].
\]

Then, using the above method, we obtain the text features \( T_{E_i}^G \) and \( T_{E_i}^{L_t} \) for the entity.
5.4 Multimodal Feature Interaction
Inspired by the multi-grained multimodal interaction approach (Luo et al., 2023), we build the feature interaction part. The multimodal feature interaction section consists of three different units. Notably, this section focuses only on introducing the functions of each unit, detailed mathematical derivations are provided in Appendix A.5.

Visual-Features Interaction (VFI). Image features of the mention \( M_j \) and the entity \( E_i \) interact separately. For feature interaction from \( M_j \) to \( E_i \), after passing through VFI:
\[
S_{V}^{M2E} = \mathrm{VFI}_{M2E}(V_{M_j}^G, V_{E_i}^G, V_{E_i}^L).
\]
The three input features are sufficiently interacted and integrated, resulting in the similarity matching score \( S_{V}^{M2E} \). Similarly, for the feature interaction from \( E_i \) to \( M_j \), the similarity score \( S_{V}^{E2M} \) can be obtained through VFI:
\[
S_{V}^{E2M} = \mathrm{VFI}_{E2M}(V_{E_i}^G, V_{M_j}^G, V_{M_j}^L).
\]
Based on this, the final visual similarity score \( S_V \) can be obtained:
\[
S_V = (S_{V}^{M2E} + S_{V}^{E2M})/2.
\]

Textual-Features Interaction (TFI). TFI computes the dot product of the normalized global features \( T_{M_j}^G \) and \( T_{E_i}^G \), yielding the text global-to-global similarity score \( S_T^{G2G} \):
\[
S_T^{G2G} = T_{M_j}^G \cdot T_{E_i}^G.
\]
To further uncover fine-grained clues within local features, TFI applies attention mechanism to capture context vector from the local features \( T_{M_j}^L \) and \( T_{E_i}^L \), producing the global-to-local similarity score \( S_T^{G2L} \) between the global feature \( T_{E_i}^G \) and the context vector:
\[
S_T^{G2L} = \mathrm{TFI}_{G2L}(T_{E_i}^G, T_{M_j}^L, T_{E_i}^L).
\]
Based on this, the final textual similarity score \( S_T \) can be obtained:
\[
S_T = (S_T^{G2G} + S_T^{G2L})/2.
\]

Cross-Modal Features Interaction (CMFI). CMFI performs a fine-grained fusion of features across modalities. It integrates visual and textual features to generate a new context vector, \( h_e \):
\[
h_e = \mathrm{CMFI}(T_{E_i}^G, V_{E_i}^L).
\]
The mention is processed similarly to produce the new context vector \( h_m \):
\[
h_m = \mathrm{CMFI}(T_{M_j}^G, V_{M_j}^L).
\]
Based on this, the final multimodal similarity score \( S_C \) can be obtained:
\[
S_C = h_e \cdot h_m.
\]

5.5 Contrastive Learning
Based on the three similarity scores \( S_V, S_T, \) and \( S_C \), the model is trained using contrastive loss function. For a mention \( M \) and entity \( E \), the combined similarity score is the average of the similarity scores from the three independent units:
\[
S(M, E) = (S_V + S_T + S_C)/3.
\]
This loss function can be formulated as:
\[
\mathcal{L}_O = -log \frac{\exp(S(M_j, E_i))}{\sum_i \exp(S(M_j, E'_i))},
\]
where \( E_i \) represents the positive entity corresponding to \( M_j \), while \( E'_i \) denotes negative entity from the knowledge base \( \mathcal{E} \). It is expected to assign higher evaluation to positive mention-entity pairs and lower evaluation to negative ones.

Similarly, the three independent units are trained separately using contrastive loss function:
\[
\mathcal{L}_X = -log \frac{\exp(S_X(M_j, E_i))}{\sum_i \exp(S_X(M_j, E'_i))}, X \in \{V, T, C\}.
\]
The final optimization objective function is expressed as:
\[
\mathcal{L} = \mathcal{L}_O + \lambda (\mathcal{L}_V + \mathcal{L}_T + \mathcal{L}_C),
\]
where \( \lambda \) is the hyperparameter to control the loss.

6 Experiments
6.1 Experimental Settings
All the training and testing are conducted on a device equipped with 4 Intel(R) Xeon(R) Platinum 8380 CPUs and 8 NVIDIA A800-SXM4-80GB GPUs. Detailed experimental settings are provided in Appendix A.1. To comprehensively evaluate the effectiveness of our approach, we compare IIER with various competitive MEL baselines and VLM baselines. A detailed introduction of these baselines is provided in the Appendix A.2.

For the VP-MEL task experiments, all approaches are evaluated on the VPWiki dataset. And for the MEL task experiments, all approaches are evaluated on the WikiDiverse (Wang et al., 2022c) dataset. Additional experiments and detailed explanations are provided in the Appendix A.10.
Table 2: Performance comparison on the VP-MEL(a) and MEL(b) tasks. Baseline results marked with "*" are based on Sui et al. (2024). Each method is run 5 times with different random seeds, and the mean value of each metric is reported. The best score is highlighted in bold. Detailed evaluation metrics can be found in Appendix A.3.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">VP-MEL</th>
    <th colspan="3">MEL</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
  </tr>
  <tr>
    <td>BLIP-2-xl (Li et al., 2023b)</td>
    <td>15.86</td>
    <td>35.41</td>
    <td>45.32</td>
    <td>34.39</td>
    <td>51.07</td>
    <td>57.83</td>
  </tr>
  <tr>
    <td>BLIP-2-xxl (Li et al., 2023b)</td>
    <td>21.90</td>
    <td>37.31</td>
    <td>49.70</td>
    <td>60.59</td>
    <td>75.59</td>
    <td>83.30</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3-7b (Ye et al., 2023)</td>
    <td>29.46</td>
    <td>30.45</td>
    <td>48.94</td>
    <td>61.21</td>
    <td>79.63</td>
    <td>85.18</td>
  </tr>
  <tr>
    <td>LLaVA-1.5-7b (Liu et al., 2024)</td>
    <td>43.20</td>
    <td>64.35</td>
    <td>65.71</td>
    <td>53.14</td>
    <td>70.93</td>
    <td>77.59</td>
  </tr>
  <tr>
    <td>LLaVA-1.5-13b (Liu et al., 2024)</td>
    <td>32.93</td>
    <td>65.56</td>
    <td>66.92</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MiniGPT-4-7b (Zhu et al., 2024)</td>
    <td>28.10</td>
    <td>33.53</td>
    <td>37.31</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MiniGPT-4-13b (Zhu et al., 2024)</td>
    <td>37.61</td>
    <td>37.61</td>
    <td>40.03</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VELML (Zheng et al., 2022)</td>
    <td>22.51</td>
    <td>37.61</td>
    <td>43.35</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GHMFC (Wang et al., 2022a)</td>
    <td>25.53</td>
    <td>41.39</td>
    <td>48.94</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MIMIC (Luo et al., 2023)</td>
    <td>24.62</td>
    <td>42.35</td>
    <td>49.25</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MELOV (Song et al., 2024)</td>
    <td>26.44</td>
    <td>42.75</td>
    <td>51.51</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>IIER(ours)</td>
    <td><b>48.36</b></td>
    <td><b>67.51</b></td>
    <td><b>77.50</b></td>
    <td><b>69.47</b></td>
    <td><b>84.43</b></td>
    <td><b>88.79</b></td>
  </tr>
</table>

6.2 Main Results

Results on VP-MEL. As shown in Table 2a, IIER significantly outperforms all other methods on VP-MEL task. First, among the VLM methods, LLaVA-1.5 has the smallest performance gap compared to our method, with differences of 5.16%, 3.16%, and 11.79% from IIER across the three metrics, respectively. Even so, given the significantly lower training cost compared to LLaVA, IIER offers a clear advantage in efficiency while achieving competitive performance. Second, there is a notable performance gap between MEL methods and IIER. MEL methods struggle with effective entity linking in scenarios where mention words are absent, underscoring their limitations and the robustness of our approach.

Results on MEL. Table 2b presents the experimental results comparing IIER with other methods on MEL dataset. During testing, the Detective-VLM analyzes image and text data to generate a concise representation of mention words, which are concatenated with the text and used for entity linking similarity calculation. With enhanced visual features and external knowledge, IIER demonstrates excellent performance in the MEL task. Although our work primarily focuses on VP-MEL rather than MEL, IIER still demonstrates strong competitiveness compared to the state-of-the-art MEL method. This highlights the effectiveness of external implicit knowledge in supporting the reasoning process of entity linking.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">WikiDiverse</th>
    <th colspan="3">WikiDiverse*</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
  </tr>
  <tr>
    <td>VELML</td>
    <td>55.53</td>
    <td>78.11</td>
    <td>84.61</td>
    <td>15.35</td>
    <td>26.32</td>
    <td>31.38</td>
  </tr>
  <tr>
    <td>GHMFC</td>
    <td>61.17</td>
    <td>80.53</td>
    <td>86.21</td>
    <td>17.37</td>
    <td>28.97</td>
    <td>34.36</td>
  </tr>
  <tr>
    <td>MIMIC</td>
    <td>63.51</td>
    <td>81.04</td>
    <td>86.43</td>
    <td>17.23</td>
    <td>29.60</td>
    <td>34.84</td>
  </tr>
  <tr>
    <td>MELOV</td>
    <td>67.32</td>
    <td>83.69</td>
    <td>87.54</td>
    <td>17.66</td>
    <td>30.03</td>
    <td>36.43</td>
  </tr>
  <tr>
    <td>IIER</td>
    <td><b>69.47</b></td>
    <td><b>84.43</b></td>
    <td><b>88.79</b></td>
    <td><b>23.87</b></td>
    <td><b>38.37</b></td>
    <td><b>45.14</b></td>
  </tr>
</table>

Table 3: Performance comparison in the absence of mention words on the WikiDiverse dataset. The symbol "*" represents the dataset without annotated mention words.

6.3 Detailed Analysis

Influence of Mention Words on MEL Methods. As shown in Table 3, the performance of MEL methods drop significantly across all three metrics in the absence of mention words. The average performance decline is 72.65%, 64.48%, and 60.28%, respectively. This indicates that MEL methods fail to extract meaningful information from visual and textual data, making them unsuitable for tasks without mention words. In contrast, even without Detective-VLM, visual prompts, or mention words, IIER can still achieve the best metrics. This demonstrates that IIER in the VP-MEL task possesses a stronger capability to leverage both image and text
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="5">VP-MEL</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
    <th>H@10</th>
    <th>H@20</th>
  </tr>
  <tr>
    <td>MiniGPT-4-7b</td>
    <td>28.55</td>
    <td>43.66</td>
    <td>52.27</td>
    <td>62.99</td>
    <td>70.70</td>
  </tr>
  <tr>
    <td>MiniGPT-4-13b</td>
    <td>27.04</td>
    <td>43.96</td>
    <td>53.02</td>
    <td>63.44</td>
    <td>70.72</td>
  </tr>
  <tr>
    <td>BLIP-2-xl</td>
    <td>37.16</td>
    <td>54.38</td>
    <td>59.52</td>
    <td>66.62</td>
    <td>72.81</td>
  </tr>
  <tr>
    <td>BLIP-2-xxl</td>
    <td>40.63</td>
    <td>54.53</td>
    <td>61.78</td>
    <td>68.73</td>
    <td>74.62</td>
  </tr>
  <tr>
    <td>LLaVA-1.5-7b</td>
    <td>42.45</td>
    <td>63.14</td>
    <td>69.03</td>
    <td>76.74</td>
    <td>82.33</td>
  </tr>
  <tr>
    <td>LLaVA-1.5-13b</td>
    <td>41.54</td>
    <td>59.37</td>
    <td>66.92</td>
    <td>73.11</td>
    <td>77.80</td>
  </tr>
  <tr>
    <td>Detective-VLM(ours)</td>
    <td><b>48.36</b></td>
    <td><b>67.51</b></td>
    <td><b>77.50</b></td>
    <td><b>82.59</b></td>
    <td><b>87.90</b></td>
  </tr>
</table>

Table 4: Performance comparison in different VLMs.

<table>
  <tr>
    <th rowspan="2">\(V^G\)-Layer</th>
    <th colspan="5">VP-MEL</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
    <th>H@10</th>
    <th>H@20</th>
  </tr>
  <tr>
    <td>Single Shallow Layer</td>
    <td>39.88</td>
    <td>60.88</td>
    <td>71.00</td>
    <td>79.31</td>
    <td>86.56</td>
  </tr>
  <tr>
    <td>Single Deep Layer</td>
    <td>39.73</td>
    <td>58.91</td>
    <td>69.94</td>
    <td>80.82</td>
    <td><b>88.07</b></td>
  </tr>
  <tr>
    <td>(Shallow+3 Deep) Layers</td>
    <td>43.66</td>
    <td>60.88</td>
    <td>68.58</td>
    <td>78.70</td>
    <td>84.29</td>
  </tr>
  <tr>
    <td>(3 Shallow+Deep) Layers</td>
    <td>40.33</td>
    <td>59.22</td>
    <td>67.37</td>
    <td>75.38</td>
    <td>83.23</td>
  </tr>
  <tr>
    <td>IIER</td>
    <td><b>48.36</b></td>
    <td><b>67.51</b></td>
    <td><b>77.50</b></td>
    <td><b>82.59</b></td>
    <td><b>87.90</b></td>
  </tr>
</table>

Table 5: Performance comparison across different feature layers in \(V^G\).

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="5">VP-MEL</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
    <th>H@10</th>
    <th>H@20</th>
  </tr>
  <tr>
    <td>IIER</td>
    <td><b>48.36</b></td>
    <td><b>67.51</b></td>
    <td><b>77.50</b></td>
    <td><b>82.59</b></td>
    <td><b>87.90</b></td>
  </tr>
  <tr>
    <td>IIER<sup>†</sup></td>
    <td>41.54</td>
    <td>59.06</td>
    <td>61.93</td>
    <td>66.62</td>
    <td>72.96</td>
  </tr>
  <tr>
    <td>IIER<sup>‡</sup></td>
    <td>35.65</td>
    <td>53.93</td>
    <td>65.26</td>
    <td>73.57</td>
    <td>80.51</td>
  </tr>
  <tr>
    <td>IIER<sup>*</sup></td>
    <td>35.03</td>
    <td>53.80</td>
    <td>65.01</td>
    <td>73.26</td>
    <td>80.39</td>
  </tr>
</table>

Table 6: The model marked "†" uses a VLM without fine-tuning. The model marked "‡" without VLM. The model marked "*" without VLM and Visual Prompts.

we choose to concatenate the deep features with the shallow features. Experimental results show that the best performance is achieved when the proportion of deep features is larger.

Ablation Study. In Table 6, we conduct ablation study on the IIER framework. Our instruction-based fine-tuning standardizes the format of the VLM-generated auxiliary information to minimize irrelevant noise. Without fine-tuning, the VLM tends to produce unstructured and often irrelevant text, which can interfere with downstream reasoning. As shown in the results, IIER exhibits significant performance degradation in H@5, H@10, and H@20 under this setting. Then we remove the Detective-VLM module from IIER, which results in a decline across all metrics. Notably, even without Detective-VLM, IIER shows robust entity linking performance, outperforming MEL methods as shown in Table 2a. This highlights the ability of IIER to efficiently leverage multimodal information from both images and text. Subsequently, removing the visual prompts from the images results in a decline across all metrics, emphasizing the crucial role of visual prompts in guiding the model to focus on relevant regions within the images. Note that the slight decrease in metrics does not suggest a diminished significance of visual prompts, as they are integral to the functioning of the VLM.

Effect Analysis of Detective-VLM. As shown in Table 4, we evaluate the effectiveness of Detective-VLM by replacing it with various VLMs and analyzing the results. Our method achieves the best performance across all metrics. In particular, Detective-VLM shows an absolute improvement of 5.91% in Hit@1 compared to the second-best approach. In contrast, non-fine-tuned VLMs often produce a large amount of irrelevant information, which hampers subsequent processing.

Contributions of Visual Features from Different Layers. As shown in Table 5, we combine visual features from different layers during the extraction of \(V^G\) to compare the effects of various combinations. In the deeper layers of CLIP visual encoder, the model tends to focus more on abstract, high-level concepts. VP-MEL focuses on aligning high-level concepts between images and text, facilitating the capture of their semantic correspondence. This explains why using a single deep layer feature achieves the highest H@20 score of 88.07%. However, in the VP-MEL task, low-level texture details are equally important. Shallow texture features need to be extracted to help the model focus on the presence of visual prompts. Based on this,

7 Conclusion

In this paper, we propose VP-MEL, a novel task designed to link visual regions in image-text pairs to their corresponding entities in a knowledge base, guided by visual prompts. To support this task, we develop VPWiki, a high-quality dataset constructed using an automated annotation pipeline to improve annotation efficiency. To tackle VP-MEL, we propose IIER, a framework that effec-
tively leverages visual prompts to extract enriched local visual features and generate supplementary textual information. IIER maintains a balance between visual and textual features, preventing excessive reliance on a single modality. Extensive experimental results demonstrate that IIER surpasses state-of-the-art methods. Furthermore, VP-MEL significantly alleviates the constraints of mention words and expands the applicability of MEL to real-world scenarios.

Limitations

VP-MEL expands the application scenarios of MEL, allowing users to directly annotate areas of interest within images. However, this requires a correlation between the image and the text. In cases where the image and text are uncorrelated, the performance of VP-MEL may degrade. In practical applications, users may utilize arbitrarily shaped regions to indicate areas of interest. Future research will aim to refine the design of visual prompts for improved adaptability and performance. We hope this work will inspire further research into leveraging recent advancements in both natural language processing and computer vision to enhance performance.

Ethics Statement

The datasets employed in this paper, WikiDiverse, WikiMEL, and RichpediaMEL, are all publicly accessible. As such, the images, texts, and knowledge bases referenced in this study do not infringe upon the privacy rights of any individual.

References

Omar Adjali, Romaric Besançon, Olivier Ferret, Hervé Le Borgne, and Brigitte Grau. 2020. Multimodal entity linking for tweets. In ECIR, pages 463–478. Springer.

Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. 2024. Vip-llava: Making large multimodal models understand arbitrary visual prompts. In CVPR, pages 12914–12923.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In HLT-NAACL, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Zhang Dongjie and Longtao Huang. 2022. Multimodal knowledge learning for named entity disambiguation. In Findings of ACL: EMNLP, pages 3160–3169, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net.

Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. 2022. An empirical study of training end-to-end vision-and-language transformers. In CVPR, pages 18166–18176.

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. In Psychological bulletin, volume 76, page 378. American Psychological Association.

Jingru Gan, Jinchang Luo, Haiwei Wang, Shuhui Wang, Wei He, and Qingming Huang. 2021. Multimodal entity linking: A new dataset and a baseline. In ACM Multimedia, page 993–1001, New York, NY, USA. Association for Computing Machinery.

Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhui Chen. 2024. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160.

Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 5583–5594. PMLR.

Jinyuan Li, Han Li, Zhuo Pan, Di Sun, Jiahao Wang, Wenkun Zhang, and Gang Pan. 2023a. Prompting chatgpt in mner: Enhanced multimodal named entity recognition with auxiliary refined knowledge. In EMNLP.

Jinyuan Li, Han Li, Di Sun, Jiahao Wang, Wenkun Zhang, Zan Wang, and Gang Pan. 2024a. LLMs as bridges: Reformulating grounded multimodal named entity recognition. In Findings of ACL: ACL, pages 1302–1318, Bangkok, Thailand. Association for Computational Linguistics.

Jinyuan Li, Ziyan Li, Han Li, Jianfei Yu, Rui Xia, Di Sun, and Gang Pan. 2024b. Advancing grounded multimodal named entity recognition via llm-based reformulation and box-based segmentation. arXiv preprint arXiv:2406.07268.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, volume 34, pages 9694–9705.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306.

Pengfei Luo, Tong Xu, Shiwei Wu, Chen Zhu, Linli Xu, and Enhong Chen. 2023. Multi-grained multimodal interaction network for entity linking. In KDD, page 1583–1594. New York, NY, USA. Association for Computing Machinery.

Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity disambiguation for noisy social media posts. In ACL (Volume 1: Long Papers), pages 2000–2008, Melbourne, Australia. Association for Computational Linguistics.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shao-han Huang, Shuming Ma, Qixiang Ye, and Furu Wei. 2024. Grounding multimodal large language models to the world. In ICLR. OpenReview.net.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR.

Senbao Shi, Zhenran Xu, Baotian Hu, and Min Zhang. 2024. Generative multimodal entity linking. In LREC-COLING, pages 7654–7665, Torino, Italia. ELRA and ICCL.

Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What does clip know about a red circle? visual prompt engineering for vlsms. In ICCV, pages 11987–11997.

Shezheng Song, Shan Zhao, Chengyu Wang, Tianwei Yan, Shasha Li, Xiaoguang Mao, and Meng Wang. 2024. A dual-way enhanced framework from text matching point of view for multimodal entity linking. In AAAI, volume 38, pages 19008–19016.

Xuhui Sui, Ying Zhang, Yu Zhao, Kehui Song, Baohang Zhou, and Xiaojie Yuan. 2024. MELOV: Multimodal entity linking with optimized visual features in latent space. In Findings of ACL: ACL, pages 816–826, Bangkok, Thailand. Association for Computational Linguistics.

Peng Wang, Jiangheng Wu, and Xiaohang Chen. 2022a. Multimodal entity linking with gated hierarchical fusion and contrastive training. In SIGIR, page 938–948, New York, NY, USA. Association for Computing Machinery.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, pages 23318–23340. PMLR.

Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Rui Wang, Ming Yan, Lihan Chen, and Yanghua Xiao. 2022c. WikiDiverse: A multimodal entity linking dataset with diversified contextual topics and entity types. In ACL (Volume 1: Long Papers), pages 4785–4797, Dublin, Ireland. Association for Computational Linguistics.

Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In EMNLP, pages 6397–6407, Online. Association for Computational Linguistics.

Shangyu Xing, Fei Zhao, Zhen Wu, Chunhui Li, Jianbing Zhang, and Xinyu Dai. 2023. Drin: Dynamic relation interactive network for multimodal entity linking. In ACM Multimedia, page 3599–3608, New York, NY, USA. Association for Computing Machinery.

Chengmei Yang, Bowei He, Yimeng Wu, Chao Xing, Lianghua He, and Chen Ma. 2023. MMEL: A joint learning framework for multi-mention entity linking. In UAI, volume 216 of Proceedings of Machine Learning Research, pages 2411–2421. PMLR.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In CVPR, pages 13040–13051.

Gongrui Zhang, Chenghuan Jiang, Zhongheng Guan, and Peng Wang. 2023a. Multimodal entity linking with mixed fusion mechanism. In DASFAA, pages 607–622. Springer.

Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. 2023b. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601.

Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al. 2023. Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning. arXiv preprint arXiv:2307.09474.

Qiushuo Zheng, Hao Wen, Meng Wang, and Guilin Qi. 2022. Visual entity linking via multi-modal learning. In Data Intel, volume 4, pages 1–19.
Chong Zhou, Chen Change Loy, and Bo Dai. 2022. Extract free dense labels from clip. In ECCV, pages 696–712. Springer.

Qiang Zhou, Chaohui Yu, Shaofeng Zhang, Sitong Wu, Zhibing Wang, and Fan Wang. 2023. Regionlibp: A unified multi-modal pre-training framework for holistic and regional comprehension. arXiv preprint arXiv:2308.02299.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR. OpenReview.net.

A Appendix

A.1 Experimental Settings

For our proposed model framework, we use pre-trained ViT-B/32 (Dosovitskiy et al., 2021) as the visual encoder, initialized with weights from CLIP-ViT-Base-Patch32\textsuperscript{1}, with \( d_v \) and \( d_c \) set to 96. The number of epochs is set to 20, and the learning rate is tuned to \( 1 \times 10^{-5} \). The batch size is set to 128. In the loss function, \( \lambda \) is set to 1. For the text encoder, we select pre-trained BERT model (Devlin et al., 2019), setting the maximum input length for text to 40 and the output feature dimension \( d_t \) to 512. Without including the VLM, the size of the trainable parameters is 153 M, and the total estimated model parameters size is 613 M. We train and test on a device equipped with 4 Intel(R) Xeon(R) Platinum 8380 CPUs and 8 NVIDIA A800-SXM4-80GB GPUs.

A.2 Descriptions of Baselines

To thoroughly evaluate the performance of our method, we compare it against strong MEL baselines, including BERT (Devlin et al., 2019), BLINK (Wu et al., 2020), JMEL (Adjali et al., 2020), VELML (Zheng et al., 2022), GHMFC (Wang et al., 2022a), MIMIC (Luo et al., 2023) and MELOV (Sui et al., 2024).

Additionally, we select robust VLMs for comparison, including BLIP-2-xl\textsuperscript{2}, BLIP-2-xxl\textsuperscript{3} (Li et al., 2023b), mPLUG-Owl3-7b\textsuperscript{4} (Ye et al., 2023), LLaVA-1.5-7b\textsuperscript{5}, LLaVA-1.5-13b\textsuperscript{6} (Liu et al., 2024), MiniGPT-4-7b\textsuperscript{7}, MiniGPT-4-13b\textsuperscript{8} (Zhu et al., 2024), ViLT (Kim et al., 2021), ALBEF (Li et al., 2021), CLIP (Radford et al., 2021), and METER (Dou et al., 2022). We reimplemented JMEL, VELML and MELOV according to the original literature due to they did not release the code. We ran the official implementations of the other baselines with their default settings.

•BERT (Devlin et al., 2019) is a pre-trained language model based on the Transformer architecture, designed to deeply model contextual information from both directions of a text, generating general-purpose word representations.

•BLINK (Wu et al., 2020) present a two-stage zero-shot linking algorithm, where each entity is defined only by a short textual description.

•JMEL (Adjali et al., 2020) extracts both unigram and bigram embeddings as textual features. Different features are fused by concatenation and a fully connected layer.

•VELML (Zheng et al., 2022) utilizes VGG-16 network to obtain object-level visual features. The two modalities are fused with additional attention mechanism.

•GHMFC (Wang et al., 2022a) extracts hierarchical features of text and visual co-attention through the multi-modal co-attention mechanism.

•MIMIC (Luo et al., 2023) devise three interaction units to sufficiently explore and extract diverse multimodal interactions and patterns for entity linking.

•MELOV (Sui et al., 2024) incorporates inter-modality generation and intra-modality aggregation.

•BLIP-2 (Li et al., 2023b) effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones.

•mPLUG-Owl3 (Ye et al., 2023) propose novel hyper attention blocks to efficiently integrate vision and language into a common language-guided semantic space, thereby facilitating the processing of extended multi-image scenarios.

\footnotetext{1https://huggingface.co/openai/clip-vit-base-patch32}
\footnotetext{2https://huggingface.co/Salesforce/blip2-flan-t5-xl-coco}
\footnotetext{3https://huggingface.co/Salesforce/blip2-flan-t5-xxl}
\footnotetext{4https://huggingface.co/mPLUG/mPLUG-Owl3-7B-240728}
\footnotetext{5https://huggingface.co/liuhaoitian/llava-v1.5-7b}
\footnotetext{6https://huggingface.co/liuhaoitian/llava-v1.5-13b}
\footnotetext{7https://drive.google.com/file/d/1RY9jY0dqgLX-o38LRumkKRh6Iatop58R/view?usp=sharing}
\footnotetext{8https://drive.google.com/file/d/1a4zLvAIoDBr-36pasffmgpvH5P7CKmpze/view?usp=share_link}
•LLaVA-1.5 (Liu et al., 2024) is an end-to-end trained large multimodal model that connects a vision encoder and an LLM for general purpose visual and language understanding.
•MiniGPT-4 (Zhu et al., 2024) aligns a frozen visual encoder with a frozen LLM, Vicuna, using just one projection layer.
•ViLT (Kim et al., 2021) commissions the transformer module to extract and process visual features in place of a separate deep visual embedder.
•ALBEF (Li et al., 2021) introduce a contrastive loss to align the image and text representations before fusing them through cross-modal attention, which enables more grounded vision and language representation learning.
•CLIP (Radford et al., 2021) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image.
•METER (Dou et al., 2022) systematically investigate how to train a fully-transformer VLP model in an end-to-end manner.

A.3 Evaluation Metrics

For evaluation, we utilize Top-k accuracy as the metric that can be calculated by the following formula:

\[
\text{Accuracy}_{top-k} = \frac{1}{N} \sum_i^N I(t_i \in y_i^k),
\]

where \( N \) represents the total number of samples, and \( I \) is the indicator function. When the receiving condition is satisfied, \( I \) is set to 1, and 0 otherwise.

A.4 Detective-VLM

Detective-VLM is based on the mplug-owl2 framework (Ye et al., 2024), with instruction fine-tuning carried out using the mplug-owl2-llama2-7b model 9.

We utilize VPWiki dataset to design the fine-tuning dataset, where \{Image\} and \{Sentence\} correspond to \( m_{v_j} \) and \( m_{s_j} \) in \( M_j \), respectively. In the fine-tuning dataset, the \{Entity Name\} corresponds to the mention words in \( M_j \) that are associated with the Visual prompt, the \{Entity Type\} is one of [Person, Organization, Location, Country, Event, Works, Misc].

9https://huggingface.co/MAGAer13/mplug-owl2-llama2-7b

A.5 Feature Interaction Formula

Visual-Features Interaction (VFI). The two similarity scores \( S_V^{M2E} \) and \( S_V^{E2M} \) in visual feature interaction are calculated using the same method. Here, we take \( S_V^{M2E} \) as an example.

\[
\overline{h}_p = \text{MeanPooling}(V_{E_i}^L),
\]
\[
h_{vc} = \text{FC}(\text{LayerNorm}(\overline{h}_p + V_{M_j}^G)),
\]
\[
h_{vg} = \text{Tanh}(\text{FC}(h_{vc})),
\]
\[
h_v = \text{LayerNorm}(h_{vg} * h_{vc} + V_{E_i}^G),
\]
\[
S_V^{M2E} = h_v \cdot V_{M_j}^G.
\]

Textual-Features Interaction (TFI). The calculation of the global-to-local similarity score \( S_T^{G2L} \) incorporates an attention mechanism as follows:

\[
Q, K, V = T_{E_i}^L W_{tq}, T_{M_j}^L W_{tk}, T_{M_j}^L W_{tv},
\]
\[
H_t = \text{softmax}(\frac{QK^T}{\sqrt{d_T}})V,
\]

where \( T_{E_i}^L W_{tq}, T_{M_j}^L W_{tk}, T_{M_j}^L W_{tv} \) are learnable matrices.

\[
h_t = \text{LayerNorm}(\text{MeanPooling}(H_t)),
\]
\[
S_T^{G2L} = \text{FC}(T_{E_i}^G) \cdot h_t.
\]

Cross-Modal Features Interaction (CMFI). CMFI performs alignment and fusion of features from different modalities.

\[
h_{et}, h_{mt} = \text{FC}_{c1}(T_{E_i}^G), \text{FC}_{c1}(T_{M_j}^G),
\]
\[
H_{ev}, H_{mv} = \text{FC}_{c2}(V_{E_i}^L), \text{FC}_{c2}(V_{M_j}^L),
\]

in which \( FC_{c1} \) is defined by \( W_{c1} \in \mathbb{R}^{d_e \times d_c} \) and \( b_{c1} \in \mathbb{R}^{d_c} \). \( FC_{c2} \) is defined by \( W_{c2} \in \mathbb{R}^{d_v \times d_c} \) and \( b_{c2} \in \mathbb{R}^{d_c} \).

\[
\alpha_i = \frac{\exp(h_{et} \cdot H_{ev}^i)}{\sum_1^{n+1} \exp(h_{et} \cdot H_{ev}^i)},
\]
\[
h_{ec} = \sum_i^n \alpha_i * H_{ev}^i, i \in [1, 2, ..., (n + 1)],
\]
\[
h_{eg} = \text{Tanh}(\text{FC}_{c3}(h_{et})),
\]

in which \( FC_{c3} \) is defined by \( W_{c3} \in \mathbb{R}^{d_e \times d_c} \) and \( b_{c3} \in \mathbb{R}^{d_c} \).

\[
h_e = \text{LayerNorm}(h_{eg} * h_{et} + h_{ec}).
\]

By replacing inputs \( h_{et} \) and \( H_{ev} \) with \( h_{mt} \) and \( H_{mv} \), \( h_m \) can be obtained using the aforementioned formula.
<table>
  <tr>
    <th></th>
    <th>WIKIDiverse</th>
    <th>WikiMEL</th>
    <th>RichpediaMEL</th>
  </tr>
  <tr>
    <td>Sentences</td>
    <td>7,405</td>
    <td>22,070</td>
    <td>17,724</td>
  </tr>
  <tr>
    <td>M. in train</td>
    <td>11,351</td>
    <td>18,092</td>
    <td>12,463</td>
  </tr>
  <tr>
    <td>M. in valid</td>
    <td>1,664</td>
    <td>2,585</td>
    <td>1,780</td>
  </tr>
  <tr>
    <td>M. in test</td>
    <td>2,078</td>
    <td>5,169</td>
    <td>3,562</td>
  </tr>
  <tr>
    <td>Entities</td>
    <td>132,460</td>
    <td>109,976</td>
    <td>160,935</td>
  </tr>
</table>

Table 7: Statistics of WIKIDiverse, WikiMEL, and RichpediaMEL. M. denotes Mentions.

A.6 Data Annotation Pipeline

Please note that the pipeline serves as a pre-processing stage for data annotation. We use the Visual Entailment Module and the Visual Grounding Module to automatically annotate visual prompts in the images. While the accuracy of the pipeline is limited—such as its difficulty in distinguishing between specific individuals when multiple people are present—it still plays a crucial role in improving annotation efficiency. Due to these limitations, manual verification and re-annotation are necessary after pipeline processing. However, for annotators, making a simple "yes or no" judgment is much easier than selecting a specific individual. As a result, even with limited accuracy, the pipeline significantly boosts the overall efficiency of the annotation process.

For the Visual Entailment Module and Visual Grounding Module, we choose OFA_{large(VE)} and OFA_{large(VG)} (Wang et al., 2022b), respectively.

![Entity type distribution of WIKIDiverse.](page_1012_1342_340_246.png)

Figure 5: Entity type distribution of WIKIDiverse.

A.7 WikiDiverse and WikiMEL

WikiDiverse is a high-quality human-annotated MEL dataset with diversified contextual topics and entity types from Wikinews, which uses Wikipedia as the corresponding knowledge base. WikiMEL is collected from Wikipedia entities pages and contains more than 22k multimodal sentences. The statistics of WIKIDiverse and WikiMEL are shown in Table 7. The entity type distribution of WIKIDiverse is illustrated in Figure 5.

During the data collection process, we select the entire WIKIDiverse dataset along with 5,000 samples from the WikiMEL dataset. Compared to WikiMEL, WIKIDiverse features more content-rich images that better represent real-world application scenarios, making it particularly suitable for meeting the requirements of the VP-MEL task in practical contexts. Consequently, WIKIDiverse constitutes the majority of the VPWiki dataset. Additionally, we integrate the knowledge bases (KBs) from both datasets, resulting in an entity set encompassing all entities in the main namespace.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">WikiMEL</th>
    <th colspan="3">RichpediaMEL</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
  </tr>
  <tr>
    <td>ViLT*</td>
    <td>72.64</td>
    <td>84.51</td>
    <td>87.86</td>
    <td>45.85</td>
    <td>62.96</td>
    <td>69.80</td>
  </tr>
  <tr>
    <td>ALBEF*</td>
    <td>78.64</td>
    <td>88.93</td>
    <td>91.75</td>
    <td>65.17</td>
    <td>82.84</td>
    <td>88.28</td>
  </tr>
  <tr>
    <td>CLIP*</td>
    <td>83.23</td>
    <td>92.10</td>
    <td>94.51</td>
    <td>67.78</td>
    <td>85.22</td>
    <td>90.04</td>
  </tr>
  <tr>
    <td>METER*</td>
    <td>72.46</td>
    <td>84.41</td>
    <td>88.17</td>
    <td>63.96</td>
    <td>82.24</td>
    <td>87.08</td>
  </tr>
  <tr>
    <td>BERT*</td>
    <td>74.82</td>
    <td>86.79</td>
    <td>90.47</td>
    <td>59.55</td>
    <td>81.12</td>
    <td>87.16</td>
  </tr>
  <tr>
    <td>BLINK*</td>
    <td>74.66</td>
    <td>86.63</td>
    <td>90.57</td>
    <td>58.47</td>
    <td>81.51</td>
    <td>88.09</td>
  </tr>
  <tr>
    <td>JMEL*</td>
    <td>64.65</td>
    <td>79.99</td>
    <td>84.34</td>
    <td>48.82</td>
    <td>66.77</td>
    <td>73.99</td>
  </tr>
  <tr>
    <td>VELML</td>
    <td>68.90</td>
    <td>83.50</td>
    <td>87.77</td>
    <td>62.80</td>
    <td>82.04</td>
    <td>87.84</td>
  </tr>
  <tr>
    <td>GHMF</td>
    <td>75.54</td>
    <td>88.82</td>
    <td>92.59</td>
    <td>76.95</td>
    <td>88.85</td>
    <td>92.11</td>
  </tr>
  <tr>
    <td>MIMIC</td>
    <td>87.98</td>
    <td>95.07</td>
    <td>96.37</td>
    <td>81.02</td>
    <td>91.77</td>
    <td>94.38</td>
  </tr>
  <tr>
    <td>MELOV*</td>
    <td>88.91</td>
    <td>95.61</td>
    <td>96.58</td>
    <td>84.14</td>
    <td>92.81</td>
    <td>94.89</td>
  </tr>
  <tr>
    <th>IIER</th>
    <th>88.93</th>
    <th>95.69</th>
    <th>96.73</th>
    <th>84.63</th>
    <th>93.27</th>
    <th>95.30</th>
  </tr>
</table>

Table 8: Baseline results marked with "*" according to Sui et al. (2024). We run each method three times with different random seeds and report the mean value of every metric. The best score is highlighted in bold.

A.8 Additional Ablation Study

The ablation study of the loss function is shown in table 9. When only \( L_O \) is computed, the model performance degrades. When the loss function consists of \( L_O \) and a single \( L_X \), the performance further declines in certain cases (e.g., \( L_O + L_T \), \( L_O + L_V \)). This degradation stems from the increased unimodal loss, which exacerbates feature interaction imbalance. When using \( L_O + L_C \), the performance improves compared to using \( L_O \) alone. This is because \( L_C \), as the loss function of the feature interaction module, alleviates the imbalance to some extent.
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="5">VP-MEL</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
    <th>H@10</th>
    <th>H@20</th>
  </tr>
  <tr>
    <td>\( L_O \)</td>
    <td>42.15</td>
    <td>56.65</td>
    <td>62.69</td>
    <td>67.67</td>
    <td>71.90</td>
  </tr>
  <tr>
    <td>\( L_O + L_T \)</td>
    <td>30.97</td>
    <td>49.85</td>
    <td>56.65</td>
    <td>63.75</td>
    <td>69.78</td>
  </tr>
  <tr>
    <td>\( L_O + L_V \)</td>
    <td>38.52</td>
    <td>54.53</td>
    <td>58.91</td>
    <td>63.60</td>
    <td>67.82</td>
  </tr>
  <tr>
    <td>\( L_O + L_C \)</td>
    <td>48.19</td>
    <td>62.54</td>
    <td>66.77</td>
    <td>69.18</td>
    <td>72.36</td>
  </tr>
  <tr>
    <td>IIER</td>
    <td><b>48.36</b></td>
    <td><b>67.51</b></td>
    <td><b>77.50</b></td>
    <td><b>82.59</b></td>
    <td><b>87.90</b></td>
  </tr>
</table>

Table 9: The loss function consists of \( L_O \) and \( L_X (X = V, T, C) \), where the similarity score in \( L_O \) is the average score of the three feature interaction modules \( L_X \).

A.9 Data Samples

We provide additional data samples categorized by entity type. The specific details can be found in Figure 7.

A.10 Additional Experiments

To comprehensively assess the performance of the IIER framework in the MEL task, we test IIER on the WikiMEL and RichpediaMEL datasets (Wang et al., 2022a). The statistics of WikiMEL and RichpediaMEL are shown in Table 7. Experimental results are shown in Table 8. The experimental results demonstrate that IIER remains highly competitive with state-of-the-art MEL method.

It is noted that within these two datasets, certain metrics of IIER exhibit values that are comparable to those of MELOV, such as H@1 and H@3 in the WikiMEL dataset. This may be attributed to the higher image quality and the homogeneous entity types (primarily *Person*) in WikiMEL and RichpediaMEL. When datasets contain fewer entity types and minimal image noise, the auxiliary information generated by IIER contributes less to performance improvement.

Nevertheless, IIER achieves the best performance on WikiDiverse, which includes a wider variety of entity types, and achieves a new SOTA for the VP-MEL task. As MEL increasingly addresses more complex scenarios, IIER shows significant potential for future advancements.

Comparative experiments with VLM2Vec (Jiang et al., 2024) are added, as presented in Table 10. Experimental results demonstrate that IIER remains state-of-the-art. While VLM2Vec performs excellently as a general-purpose information retrieval model for multimodal search, it is not specifically designed for fine-grained entity-level alignment - the core requirement of entity linking tasks. This inherent limitation partially explains VLM2Vec’s suboptimal performance on MEL tasks.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="5">VP-MEL</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@5</th>
  </tr>
  <tr>
    <td>BLIP-2-xl (Li et al., 2023b)</td>
    <td>15.86</td>
    <td>35.41</td>
    <td>45.32</td>
  </tr>
  <tr>
    <td>BLIP-2-xxl (Li et al., 2023b)</td>
    <td>21.90</td>
    <td>37.31</td>
    <td>49.70</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3-7b (Ye et al., 2023)</td>
    <td>29.46</td>
    <td>30.45</td>
    <td>48.94</td>
  </tr>
  <tr>
    <td>LLaVA-1.5-7b (Liu et al., 2024)</td>
    <td>43.20</td>
    <td>64.35</td>
    <td>65.71</td>
  </tr>
  <tr>
    <td>LLaVA-1.5-13b (Liu et al., 2024)</td>
    <td>32.93</td>
    <td>65.56</td>
    <td>66.92</td>
  </tr>
  <tr>
    <td>MiniGPT-4-7b (Zhu et al., 2024)</td>
    <td>28.10</td>
    <td>33.53</td>
    <td>37.31</td>
  </tr>
  <tr>
    <td>MiniGPT-4-13b (Zhu et al., 2024)</td>
    <td>37.61</td>
    <td>37.61</td>
    <td>40.03</td>
  </tr>
  <tr>
    <td>VELML (Zheng et al., 2022)</td>
    <td>22.51</td>
    <td>37.61</td>
    <td>43.35</td>
  </tr>
  <tr>
    <td>GHMFC (Wang et al., 2022a)</td>
    <td>25.53</td>
    <td>41.39</td>
    <td>48.94</td>
  </tr>
  <tr>
    <td>MIMIC (Luo et al., 2023)</td>
    <td>24.62</td>
    <td>42.35</td>
    <td>49.25</td>
  </tr>
  <tr>
    <td>MELOV (Song et al., 2024)</td>
    <td>26.44</td>
    <td>42.75</td>
    <td>51.51</td>
  </tr>
  <tr>
    <td>VLM2Vec (Jiang et al., 2024)</td>
    <td>44.09</td>
    <td>65.76</td>
    <td>70.01</td>
  </tr>
  <tr>
    <td>IIER(ours)</td>
    <td><b>48.36</b></td>
    <td><b>67.51</b></td>
    <td><b>77.50</b></td>
  </tr>
</table>

Table 10: Performance comparison in different VLMs.

A.11 Case Study

To clearly demonstrate the proposed VP-MEL task and the IIER model, we conduct case studies and compare them against two strong competitors (*i.e.*, LLaVA-1.5 and MIMIC), in Figure 6. As shown in Figure 6a, in the first case, all three methods correctly predicted the entity. IIER makes full use of both image and text information, allowing it to more effectively distinguish between the different individuals in the image. LLaVA-1.5 may be overwhelmed by the textual information, while MIMIC struggles to identify the correct entity when the mention words are unavailable. In the second case, both LLaVA-1.5 and MIMIC retrieve *Endeavour* as the first choice. Only IIER, with the guidance of Visual Prompts and integration of textual information, correctly predicts the right entity. In Figure 6b, we present the failed predictions. In the first case, when the content of the image interferes with the visual prompt, it impairs the reasoning process of IIER. The red box in the image bears a high similarity to the visual prompt. As a result, IIER incorrectly focuses on the wrong region of the image, ranking *Donald Trump* first. When IIER encounters difficulties in distinguishing the objects within the visual prompts, it leads to incorrect inferences. For example, in the second case, the distinguishing features of the two individuals in the image
<table>
  <tr>
    <th>Input</th>
    <th>Ground Truth Entity</th>
    <th>VP-MEL (IIER)</th>
    <th>LLaVA-1.5</th>
    <th>MEL (MIMIC)</th>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_120_320_246.png" alt="Couples together in Marvel Cinematic Universes Iron Man (left) and Spider-Man (right)."/>
    </td>
    <td rowspan="2">
      Q79037 Spider-Man fictional character in Marvel Comics
    </td>
    <td>
      TOP-1 Q79037 Spider-Man<br>
      TOP-2 Q496817 Spider-Man in other media<br>
      TOP-3 Q90307 Iron Man's armor
    </td>
    <td>
      TOP-1 Q79037 Iron Man's armor<br>
      TOP-2 Q79037 Spider-Man<br>
      TOP-3 Q412878 Marvel Cinematic Universe
    </td>
    <td>
      TOP-1 Q79037 Iron Man's armor<br>
      TOP-2 Q496817 Spider-Man in other media<br>
      TOP-3 Q79037 Spider-Man
    </td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_370_320_246.png" alt="The crew of Endeavour's final mission STS-134, which launched on Monday at 8:56 AM EDT."/>
    </td>
    <td rowspan="2">
      Q478803 STS-134 25th and last spaceflight of Space Shuttle Endeavour
    </td>
    <td>
      TOP-1 Q478803 STS-134<br>
      TOP-2 Q329037 STS-133<br>
      TOP-3 Q460468 STS-132
    </td>
    <td>
      TOP-1 Q478803 Endeavour<br>
      TOP-2 Q478803 STS-134<br>
      TOP-3 Q509090 TS-135
    </td>
    <td>
      TOP-1 Q478803 Endeavour<br>
      TOP-2 Q478803 STS-134<br>
      TOP-3 Q509090 TS-135
    </td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(a) Successful predictions.

<table>
  <tr>
    <th>Input</th>
    <th>Ground Truth Entity</th>
    <th>VP-MEL (IIER)</th>
    <th>LLaVA-1.5</th>
    <th>MEL (MIMIC)</th>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_620_320_246.png" alt="2016: that year, US president Donald Trump was named Time 'Person of the Year'."/>
    </td>
    <td rowspan="2">
      Q43297 Time American magazine and website
    </td>
    <td>
      TOP-1 Q22696 Donald Trump<br>
      TOP-2 Q367926 Time Person of the Year<br>
      TOP-3 Q43297 Time
    </td>
    <td>
      TOP-1 Q43297 Time<br>
      TOP-2 Q22696 Donald Trump<br>
      TOP-3 Q367926 Time Person of the Year
    </td>
    <td>
      TOP-1 Q43297 Time<br>
      TOP-2 Q22696 Donald Trump<br>
      TOP-3 Q367926 Time Person of the Year
    </td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_870_320_246.png" alt="Ozzy and Sharon Osbourne visit the USS Missouri on March 5, 2006."/>
    </td>
    <td rowspan="2">
      Q180685 Sharon Osbourne British-American television personality
    </td>
    <td>
      TOP-1 Q313151 Ozzy Osbourne<br>
      TOP-2 Q180685 Sharon Osbourne<br>
      TOP-3 Q199411 USS - Missouri
    </td>
    <td>
      TOP-1 Q313151 Ozzy Osbourne<br>
      TOP-2 Q180685 Sharon Osbourne<br>
      TOP-3 Q199411 USS - Missouri
    </td>
    <td>
      TOP-1 Q199412 USS Missouri<br>
      TOP-2 Q272560 USS Missouri<br>
      TOP-3 Q199412 USS Missouri
    </td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(b) Failed predictions.

Figure 6: Case study for VP-MEL. Each row is a case, which contains Input, ground truth entity, and top three retrieved entities of three methods, i.e., IIER (ours), LLaVA-1.5 (Liu et al., 2024), MIMIC (Luo et al., 2023). Each retrieved entity is described by its Wikidata QID and entity name, with the entity marked with a checkmark indicating the correct one.

are obstructed, which causes IIER to struggle in differentiating between them. The image content in real-world data is often complex, which makes VP-MEL a challenging task. We hope that this task can be further refined and developed over time.
<table>
  <tr>
    <th>Type</th>
    <th>Image of mention</th>
    <th>Text of mention</th>
    <th>Image of entity</th>
    <th>Entity</th>
  </tr>
  <tr>
    <td><b>Person</b></td>
    <td>Australia's Douglas Utejesenovic in action against East German Eberhard Vogel at the 1974 FIFA World Cup, Australia men's first World Cup appearance.</td>
    <td></td>
    <td></td>
    <td>
      <pre>
        "qid": "Q681184"
        "entity_name": "Eberhard Vogel"
        "desc": "German footballer"
      </pre>
    </td>
  </tr>
  <tr>
    <td><b>Organization</b></td>
    <td>A restaurant in Exeter in Devon, UK.</td>
    <td></td>
    <td></td>
    <td>
      <pre>
        "qid": "Q38896"
        "entity_name": "McDonald's"
        "desc": "American fast food restaurant chain"
        ...
      </pre>
    </td>
  </tr>
  <tr>
    <td><b>Location</b></td>
    <td>Map showing San Fernando within the province of Romblon.</td>
    <td></td>
    <td></td>
    <td>
      <pre>
        "qid": "Q13875"
        "entity_name": "Romblon"
        "desc": "province of the Philippines"
        ...
      </pre>
    </td>
  </tr>
  <tr>
    <td><b>Country</b></td>
    <td>President Bush with PM Tymoshenko in 2008.</td>
    <td></td>
    <td></td>
    <td>
      <pre>
        "qid": "Q80"
        "entity_name": "United States of America"
        "desc": "country in North America"
        ...
      </pre>
    </td>
  </tr>
  <tr>
    <td><b>Event</b></td>
    <td>First place winner Brazilian Teresinha Guilhermina and her guide Antônio de Santana across the line in the women's 200 m final T11 is underway.</td>
    <td></td>
    <td></td>
    <td>
      <pre>
        "qid": "Q21155"
        "entity_name": "200 metres"
        "desc": "sprint running event"
        ...
      </pre>
    </td>
  </tr>
  <tr>
    <td><b>Works</b></td>
    <td>Voice actor Rob Paulson tries to find the right words for Pinky during the Masquerade.</td>
    <td></td>
    <td></td>
    <td>
      <pre>
        "qid": "Q1580736"
        "entity_name": "Pinky and the Brain"
        "desc": "animated television series"
        ...
      </pre>
    </td>
  </tr>
  <tr>
    <td><b>Misc</b></td>
    <td>A California owl in Redwoods Park in California.</td>
    <td></td>
    <td></td>
    <td>
      <pre>
        "qid": "Q748821"
        "entity_name": "Spotted Owl"
        "desc": "species of bird"
        ...
      </pre>
    </td>
  </tr>
</table>

Figure 7: Examples of the VPWiki dataset. Each row represents a sample corresponding to a specific entity type, which contains the entity type, image of mention, text of mention, image of entity, and entity.

Reviewer paper summaries: The paper proposes a new task, VMEL, for MEL and introduces the corresponding dataset. Extensive experiments are conducted. However, there are still issues regarding the motivation for introducing this task, and the proposed method lacks innovation, as detailed in the Weakness section.

This paper introduces a new task, namely visual prompt guided multimodal entity linking. The task is aimed to solve the problem that the existing MEL tasks often focus more on the mention words and neglect the visual regions. To fill this gap, the new task annotates mention on the images to enhance the effectiveness of multimodal entity linking.

This paper introduce a new task, VP-MEL,aiming to link visual regions in image-text pairs to the corresponding entities in knowledge base, guided by visual prompts. To support this task, this paper construct a high-quality dataset, VPWiki, leveraging an automated annotation pipeline to enhance the efficiency of data annotation. To address the VP-MEL task, this paper propose FBMEL, a framework that can effectively utilize visual prompts to extract enhanced local visual features and generate supplementary textual information.
Summary of strengths: This paper addresses the reliance on mention words in MEL, proposing a new task called VMEL, which is based on performing MEL on image mentions.
The paper provides extensive experimental validation.

1. The proposed task broadens the application of multimodal entity linking in the real word scenarios. The annotation of image regions could make the entity linking task more complete.

2. The proposed dataset is usable in the entity linking task which allows a specific region to a specific entity in the knowledge base.

3. The authors propose a new framework to solve this task and provide comprehensive experimental results.

1. The motivation of the study is well-defined, and the paper demonstrates a high level of readability. Both the motivation and the methodology are clearly and effectively presented.  
2. The experimental results are satisfactory, with comparative experiments confirming the effectiveness of the proposed method.
Summary of weaknesses: - In L51, the paper mentions that mentions are strong cues for linking, so isn’t the high similarity between mention images and entity images also a strong cue?
- In L54, the paper states, "Although some ambiguity remains..." When a task has low ambiguity, it doesn’t make sense to remove the mention and claim the ambiguity as a reason for a new, useful task.
- The first and second innovations in the paper essentially refer to the same thing.
- In L427, the loss weights for V, T, and C are not considered, and this is not addressed in the subsequent ablation study.
- The ablation study in Table 6 shows that VLM has the most significant impact, while visual prompts have minimal effect on the results, making it difficult to justify the effectiveness of visual prompts. On the other hand, the VLM mentioned in the paper seems to involve fine-tuning the format using LLM to identify entity names and types. So, why not directly use LLM to handle the entire MEL process? The VLM part, which shows the most impact in the ablation study, lacks innovation.
- "Global" and "local" approaches are widely discussed. The "global" approach in this paper involves using VLM to detect potential entity types and names from the image, which seems to lack novelty.
- The issue of image modality impurity is mentioned in the introduction, but it is not adequately addressed in the paper. Even with VLM performing type and name detection, the method still suffers from image modality impurity.
- The logic is somewhat weak. In L64, the statement "This excessive reliance on mention words confines MEL to a text-focused task" does not seem like a flaw or an issue that needs addressing. It's also hard to derive "It is necessary to introduce a new MEL task to overcome such a constraint" from this. The VMEL task proposed in this paper is essentially a VQA task targeting persons. However, the purpose of MEL is to better integrate or relate existing knowledge to a knowledge base (KB), which is not significantly advanced by the proposed VMEL task. The issue raised in L92 seems unnecessary.
- Based on the experimental results, the paper shows significant performance improvements on the proposed VMEL task. However, according to Table 8, the FBMEL is weaker than the MIMIC and MELOV models. Furthermore, the paper does not compare with other MEL models in terms of performance.
- I did not understand which dataset MEL refers to in Table 2.b. The paper does not achieve SOTA performance on this dataset.
- Why is there no comparison with LLM performance on the VMEL task? LLMs have a significant advantage in MEL tasks.

1. Though the new task is significant, the illustration of the motivation is unclear. For example, in figure, it is confusing that why MEL would extract Lolo from the sample. In the abstract you said the traditional MEL focus only on texts/mention words, but it does not mean MEL will extract unrelated entity from the text. I believe you could give a better example such as 'Ted' does not appear in the text but is shown in the image.

2. The appendix does not provide enough examples of the dataset. I'm wondering what is look like of the work and event as shown in figure 3 a.

1. The feasibility and reliability of the experiments are beyond question. However, the main concern lies in the statement "Each method is run three times." It is recommended to increase the number of runs to enhance the robustness of the results.  
2. The paper lacks discussion on relevant research and theoretical analysis concerning the use of object detection to identify objects and match them with a knowledge base.
Comments, suggestions and typos: - All figures in the paper are not vector-based, resulting in blurry images. This is likely also the reason why the PDF size is as large as 30MB.
- The images of the characters are either compressed or stretched.
- The term "mention" in the paper seems to refer to "mention image," but it is often used ambiguously to refer to "mention," which could cause confusion.
- The font size in Figure 3.b is too large, and the percentages in Figure 3.a overlap.

See weaknesses.

see Summary of Weaknesses
Overall assessment scores: 1.5

2.5

2.5
Metareview score: 2
Metareview: This paper introduces a new task, visual prompt guided multimodal entity linking (VP-MEL), which aims to link visual regions in image-text pairs to the corresponding entities in the knowledge base, guided by visual prompts. To support this task, this paper constructs a high-quality dataset, VPWiki, leveraging an automated annotation pipeline to enhance the efficiency of data annotation. The authors propose an FBMEL framework to effectively utilize visual prompts and extract enhanced local visual features and generate supplementary textual information.
Summary of reasons to publish: 1. The new task is significant, which address limitations of traditional MEL methods in scenarios without textual cues.

2. Th proposed model is easy follow and the experiments are sufficient.