MELOV: Multimodal Entity Linking with Optimized Visual Features in Latent Space (2024)

Abstract: Multimodal entity linking (MEL), which aligns ambiguous mentions within multimodal contexts to referent entities from multimodal knowledge bases, is essential for many natural language processing applications. Previous MEL methods mainly focus on exploring complex multimodal interaction mechanisms to better capture coherence evidence between mentions and entities by mining complementary information. However, in real-world social media scenarios, vision modality often exhibits low quality, low value, or low relevance to the mention. Integrating such information directly will backfire, leading to a weakened consistency between mentions and their corresponding entities. In this paper, we propose a novel latent space vision feature optimization framework MELOV, which combines inter-modality and intra-modality optimizations to address these challenges. For the inter-modality optimization, we exploit the variational autoencoder to mine shared information and generate text-based visual features. For the intra-modality optimization, we consider the relationships between mentions and build graph convolutional network to aggregate the visual features of semantic similar neighbors. Extensive experiments on three benchmark datasets demonstrate the superiority of our proposed framework.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 816–826 August 11-16, 2024 ©2024 Association for Computational Linguistics
Multimodal entity linking (MEL), which aligns ambiguous mentions within multimodal contexts to referent entities from multimodal knowledge bases, is essential for many natural language processing applications. Previous MEL methods mainly focus on exploring complex multimodal interaction mechanisms to better capture coherence evidence between mentions and entities by mining complementary information. However, in real-world social media scenarios, vision modality often exhibits low quality, low value, or low relevance to the mention. Integrating such information directly will backfire, leading to a weakened consistency between mentions and their corresponding entities. In this paper, we propose a novel latent space vision feature optimization framework MELOV, which combines inter-modality and intra-modality optimizations to address these challenges. For the inter-modality optimization, we exploit the variational autoencoder to mine shared information and generate text-based visual features. For the intra-modality optimization, we consider the relationships between mentions and build graph convolutional network to aggregate the visual features of semantic similar neighbors. Extensive experiments on three benchmark datasets demonstrate the superiority of our proposed framework.

heading: 1 Introduction
text: Entity linking (EL) is the task of assigning ambiguous mentions in text to their corresponding entities in a knowledge base. As a bridge between unstructured content and structured knowledge bases, EL plays a vital role in many downstream natural language processing applications, such as content analysis (Huang et al., 2018), semantic search (Blanco et al., 2015), question answering (Xiong et al., 2019; Longpre et al., 2021) and so on.
*Corresponding author.
Michael Jackson
Michael Kors
…
Multimodal KB
Michael Jordan
Scottie Pippen
Low-Quality
Every show of Michael was perfect.
Low-Value
Low-Relevance
Good Image
Mentions with Context
Michael led the Bulls to win the NBA
championship again.
I am very happy to buy my favorite handbag
in Michael. Scottie played very
well in Bulls. …
Multimodal KB
(a) Low-Quality
Every show of Michael was perfect.
(b) Low-Value
(c) Low-Relevance
(d) Good Image
Mentions with Context
Michael led the Bulls to win the NBA
championship again.
I am very happy to buy my favorite handbag
in Michael. Scottie played very well in Bulls.
Michael Jackson
Michael Kors
Michael Jordan
Figure 1: Examples of multimodal entity linking with low-quality, low-value, low-relevance and good visual images. Entities that share the same color as mentions are the corresponding gold entities.
Traditional EL methods (Chisholm and Hachey, 2015; Eshel et al., 2017) mainly focus on addressing the text-based EL by resolving text context. However, with the rapid development of social media, images along with text have become the most common form of web information, which brings new challenges for EL models to effectively integrate visual information and understand multimodal content. Thus, the multimodal entity linking (MEL) task (Moon et al., 2018) has been proposed, which extends the scope from textual EL to heterogeneous formats, i.e. linking mentions with textual context and visual context to referent entities in multimodal knowledge bases.
Multiple previous MEL works (Adjali et al., 2020; Gan et al., 2021; Wang et al., 2022a; Zhang and Huang, 2022; Wang et al., 2023; Xing et al., 2023; Shi et al., 2023; Luo et al., 2023; Zhang et al., 2023) mainly focus on exploring complex multimodal interaction mechanisms to better capture coherence evidence between mentions and entities by mining complementary information. Although
816
these works have achieved promising performance in the past few years, they usually assume that each mention is associated with high-quality, high-value, and high-relevance visual images, which is unrealistic in real-world social media scenarios. Figure 1 shows several social media examples and MEL still faces the pervasive challenges presented by the following poor visual images:
Low-quality. User-posted images may appear blurry, unclear, or pixelated due to factors such as image compression, low original image quality and network transmission issues.
Low-value. As shown in Figure 1(b), this image is widely used in social media, which only expresses the happy mood of users in most cases. However, this type of images does not enhance the overall comprehension of the text and may even mislead the MEL model into linking “Michael” to a person rather than the brand “Michael Kors”.
Low-relevance. Accompanying images often correspond to the whole context of text, rather than being specifically tied to the mention. Figure 1(c) is such an example whose image is low-relevance to the mention “Michael” in text, which will interfere with MEL models matching “Michael Jordan”.
There is a huge gap between all of these poor visual images and the images of their corresponding entities. Integrating them directly will backfire, leading to a weakened consistency between mentions and their corresponding entities. Some previous works (Zhang et al., 2021) have realized the negative impact of noisy images. They propose a straightforward method to remove these poor images by assessing the correlation score between the mention image and mention text. However, the low-quality and low-relevance images still possess valuable implicit visual cues. Discarding them outright would result in the loss of significant complementary visual information. Thus, while the straightforward approach is effective in handling low-value images, it is still not ideal for dealing with low-quality and low-relevance images.
To address these challenges, in this paper, we propose a novel Multimodal Entity Linking framework with Optimized Visual features in latent space, MELOV in short. Our MELOV contains two perspectives of optimization, i.e. intermodality and intra-modality. Specifically, for the inter-modality optimization, we exploit variational autoencoder to mine shared semantic information from heterogeneous textual features and generate latent vision-specific features. For the intra-
modality optimization, we observe that similar or related mentions often possess related visual information, which is very useful for optimizing poor images. An example is shown in Figure 1(d), “Michael Jordan” even appears directly in this image. To effectively aggregate the visual information of semantic similar neighbors, we construct mention graphs using correlation scores between mentions and employ graph convolutional network for information propagation within the graphs. Finally, our MELOV adaptively assesses the contributions of original visual features, inter-modality generated visual features, and intra-modality aggregated visual features to the MEL task, and fuses them in varying proportions within the latent space. The optimized visual features can simultaneously handle all types of poor images.
The main contributions of this paper are summarized as follows:
• For the first time, we analyze various types of poor images in multimodal entity linking and propose to optimize visual features in latent space, aiming to not only eliminate the negative effects of poor images but also retain the implicit visual cues of original images.
• We propose a joint optimization framework that incorporates inter-modality generation and intra-modality aggregation, effectively leveraging the shared information from heterogeneous textual features and relevant visual details of semantic similar neighbors.
• We compare our MELOV with state-of-the-art MEL approaches on three public benchmark datasets. Experimental results demonstrate the superiority of our proposed framework.

heading: 2 Related Work

heading: 2.1 Entity Linking
text: Traditional entity linking (EL) mainly focuses on text-only corpus, which has been widely explored in recent years. EL methods can be roughly divided into two groups based on their granularity: locallevel and global-level. The local-level approaches (Francis-Landau et al., 2016; Cao et al., 2017; Eshel et al., 2017; Gupta et al., 2017; Peters et al., 2019; Wu et al., 2020; Sui et al., 2022, 2023) separately link each mention according to the similarity between mentions with context and entities. The global-level approaches (Le and Titov, 2018; Yang
et al., 2018; Fang et al., 2019; Yang et al., 2019) strive to jointly disambiguate mentions by considering the global coherence of entities within the same document. While these methods have achieved significant progress, they are designed solely for text modality and cannot effectively integrate abundant complementary information from visual images. This limits the performance of EL methods in the surge of multimodal information and motivates researchers to study multimodal entity linking.

heading: 2.2 Multimodal Entity Linking
text: Multimodal entity linking (MEL) is an extension of EL that utilizes additional visual images to aid in disambiguating mentions. Moon et al. (2018) first proposes the task and leverages cross-modal attention to fuse features. Adjali et al. (2020) designs concatenation operation and triplet loss to interact modality. Wang et al. (2022a) proposes a hierarchical multimodal co-attention to mine fine-grained relationships across modalities. Zhang et al. (2023) combines global and bottleneck fusions to integrate multimodal information. Shi et al. (2023) leverages the in-context learning of LLMs to generate target entities. Xing et al. (2023) explicitly models and dynamically selects different alignment relations between mentions and entities. Luo et al. (2023) uses multi-grained interactions and unit-consistent learning to enhance the representations.
In general, all these methods mainly focus on exploring complex multimodal interaction mechanisms to better capture coherence evidence between mentions and entities. However, they overlook the prevalence of poor visual images accompanying text for MEL. Integrating them directly will backfire, leading to a weakened consistency between mentions and their corresponding entities. To re-
move the negative impact of noisy images, Zhang et al. (2021) assesses the correlation scores between the text and image of mentions to filter out poor images. However, many poor images still possess valuable implicit visual cues. Discarding them outright would result in the loss of significant complementary visual information. This motivates us to find another way, that is optimizing visual features in latent space to not only eliminate the negative effects of poor images but also retain the implicit visual cues of original images.

heading: 3 Methodology
text: Figure 2 shows the overall architecture of our MELOV. We first extract both global and local features of textual and visual inputs to obtain global descriptive semantics and preserve fine-grained details of words or image patches. Afterwards, we optimize visual features at two perspectives: intermodality generation based on the shared information of heterogeneous textual modality and intramodality aggregation based on the correlation of mentions. We obtain the optimized visual features by adaptively fusing the two perspective features and original features. Finally, we employ multigrained interactions using optimized visual features as input to derive the similarity matching score for each mention-entity pair to make the ultimate multimodal entity linking decisions.

heading: 3.1 Feature Extraction
text: To extract meaningful textual features, following Luo et al. (2023), we utilize pre-trained BERT (Devlin et al., 2019) as our textual encoder. The inputs of each mention mi and entity ei are respectively constructed as: [CLS] me [SEP] and [CLS]
en [SEP], where me = mention [SEP] sent and en = title [SEP] attr. The mention, sent, title, and attr refer to word-piece tokens of the mention, the sentence where the mention is located, the entity title, and the attributes associated with the entity. Then we feed the inputs into BERT and obtain the hidden states of all word tokens TMi and TEi . Finally, we regard the hidden state corresponding to the position of special [CLS] token as global textual features tGMi and t G Ei
, the entire hidden states as local textual features TLMi and T L Ei
. To capture expressive visual features, following Luo et al. (2023), we employ pre-trained Vision Transformer (ViT) (Dosovitskiy et al., 2021) as our visual encoder. Given an image of a mention or entity, we first rescale it into C×H×W pixels and reshape it into H × W/P 2 flattened 2D patches. Here, C represents the number of channels, H×W denotes the image resolution and P is the patch size. Then, we feed them into ViT and obtain the hidden states of all patches. Similar to the textual feature extraction, we regard the hidden state of the [CLS] as global visual features vGOMi or v G Ei
, the whole hidden states as local visual features V LOMi or V L Ei .

heading: 3.2 Visual Feature Optimization
text: In multimodal entity linking, heterogeneous textual features encompass abundant shared semantic information of visual and textual modalities, such as the characteristics and categories of mentions. On the other hand, visual features of similar or related mentions contain rich vision-specific details associated with the mention. Both the two perspectives are significant for optimizing visual features.

heading: 3.2.1 Inter-Modality Generation
text: For inter-modality optimization, we propose to utilize the cross-reconstruction of variational autoencoder (VAE) (Yi et al., 2023) to mine shared semantic information from heterogeneous textual features and generate vision-specific features in latent space. Specifically, the VAE encoder takes the mention global textual features tGMi as input to feed-forward neural networks (FFNNs) to derive latent mean vector µ and standard deviation vector σ: µ = tGMiWµ and σ = t G Mi
Wσ, where Wµ and Wσ ∈ Rdt×dz are trainable weights, dt is the dimension of textual features. The two vectors jointly describe the distribution of latent space z ∼ q(z|tGMi) = N (µ, σ
2). We utilize the reparameterization strategy (Kingma and Welling, 2014) to sample the latent variation: z = µ+σ⊙ϵ,
where ϵ ∼ N (0, I). Then the latent variation z is fed into the VAE decoder to generate global visual features vGGMi = zWd, where Wd ∈ R
dz×dv and dv is the dimension of visual features. Based on the evidence lower bound function (Kingma and Welling, 2014) for VAE, the training loss of our global feature generation is computed as follows:
LGG = ||vGOMi − v GG Mi ||2 + KL(q(z|tGMi)||p(z)) (1)
By constructing visual features as accurately as possible, our model learns to mine shared semantic information from heterogeneous textual features within latent space, enabling the generation of textbased vision-specific features. Similarly, for local features, we also generate V LGMi by utilizing VAE and calculate the local loss LLG. The final loss of our inter-modality generation is calculated as:
LG = LGG + LLG (2)

heading: 3.2.2 Intra-Modality Aggregation
text: To effectively leverage intra-modality related visual information of semantic similar neighbors, we propose to construct mention graphs and employ graph convolutional network (GCN) (Kipf and Welling, 2017) for information aggregation. Specifically, considering that the textual modality provides the most reliable information about mentions, we calculate the similarity between mentions by utilizing textual representations. The common approach for this is to calculate the Cosine or Euclidean distance. However, these traditional similarity measures are not sufficient in accurately capturing the local manifold structure and are unable to capture complex relationships such as higher-order statistics. Inspired by Kang et al. (2017); Zhang et al. (2022), we utilize a kernel-driven approach to calculate the similarity for two given mentions mi and mj :
sim(mi,mj) = exp(− ||tGMi − t G Mj ||22 2b2 ) (3)
where b represents the bandwidth used to regulate the emphasis given to the similarity of small distances relative to large distances. We determine this value by setting it as a fraction of the average distance between mentions. For all mentions in the training set, the similarity matrix A ∈ Rn×n is created by assessing the sim(mi,mj) of each pair of mentions, where n is the number of mentions.
In our similarity matrix, all cells are always positive, indicating that all mentions are treated as
similar. However, this will introduce noise from dissimilar mentions. Thus, we employ a learnable threshold to filter out these dissimilar mentions:
Aij = { Aij ifAij > τ 0 ifAij ≤ τ
(4)
where τ is the learnable threshold. We formulate the original global visual features of all mentions vGOM ∈ Rn×dv as a graph, with the similarity matrix A as the graph adjacency matrix. Then, we employ GCN to facilitate the information propagation within the graph, thereby aggregating visual information from semantic similar neighbors. A GCN layer is a nonlinear transformation which maps from vlM to v l+1 M , defined as:
vl+1M = ϕ(Av l MWg + bg) (5)
where l is the current layer index, v0M = v GO M , Wg ∈ Rdv×dv and bg ∈ Rdv , ϕ is a non-linear activation function. After L layers of GCN, we obtain the intra-modality aggregated global visual features vGAM . Similarly, for the local features, we formulate V LOM as a graph and feed them into another GCN to obtain intra-modality aggregated local visual features V LAM .

heading: 3.2.3 Information Adaptive Fusion
text: Now we possess three different types of visual features: original visual features focus on original visual images themselves, inter-modality generated visual features emphasize shared semantic information from heterogeneous textual features, and intra-modality aggregated visual features highlight related visual details of semantic similar neighbors. To take all of them into consideration, we utilize the attention mechanism to adaptively assess their contributions to the MEL task and fuse them in different proportions. Specifically, for global visual features, we put them into different FFNNs and utilize the Sigmoid function to obtain the contribution weights αO, αG and αA. Furthermore, we add a constraint αO + αG + αA = 1 by calculating αO = α O
αO+αG+αA , αG = α
G
αO+αG+αA ,
αA = 1− αO − αG. The optimized global visual features are obtained by calculating the weighted sum of different global visual features:
vGM = α O · vGOM + αG · vGGM + αA · vGAM (6)
Similarly, we also utilize the attention mechanism to obtain the optimized local visual features V LM .

heading: 3.3 Multimodal Entity Linking
text: To derive matching scores for each mention-entity pair, following Luo et al. (2023), we utilize multigrained multimodal interaction, which contains text-based, vision-based and cross-modal interactions. Our text-based interaction involves two scores. The global-to-global score measures the global consistency by using dot product: SG2GT = tGM · tGE . And the global-to-local score captures fine-grained context consistency clues between entity global features and mention context vector: SG2LT = t G E · ht, where the mention context vector ht is obtained using attention mechanism, mean pooling MP and layer norm LN: ht = LN(MP(softmax(QK
T √ dt )V )), Q = TLE , K = V =
TLM . We obtain the text-based matching score by averaging the two scores: ST = (SG2GT +S G2L T )/2.
In our vision-based interaction, we employ a dual-gated mechanism to consider both mention and entity views. For entity-to-mention interaction, we first interact entity global and mention local features: hvc = LN(vGE + MP(V L M ))Wv1, where Wv1 ∈ Rdv×dv . Then, we interact mention global features with a gate operation: hv = LN(Tanh(hvcWv2) · hvc + vGM ), where Wv2 ∈ Rdv×1. After sufficient interaction and fusion, we obtain the matching score from entities to mentions SE2MV = hv · V GE and similarly from mentions to entities SM2EV . The vision-based matching score is the average of them: SV = (SE2MV + S M2E V )/2.
Our cross-modal interaction measures the consistency between mentions and entities after integrating visual and textual information. We first use FFNNs to map the textual and visual features tGE , t G M , V L E and V L M to shared dense space features het, hmt, Hev and Hmv. To aggregate entity image patch information, we employ correlation scores between textual and visual features as guidance: αic =
exp(het·Hiev)∑np i=1 exp(het·Hiev)
, hec = ∑np
i=1 α i c·H iev, where np is the patch number. Then, we interact the textual features with a gate operation to obtain the entity cross-modal representation he = LN(Tanh(hetWc) · het + hec), where
Wc ∈ Rdc×dc . Similarly, we obtain the mention representation hm by replacing het and Hev with hmt and Hmv. We compute the cross-modal matching score using the dot product: SC = he · hm.
The final score is defined as the average of the three scores S = (ST + SV + SC)/3. To maximize the score of the corresponding entity among others and ensure to learn good representations in each interaction, we employ an in-batch contrastive training approach that considers all interactions as our loss function. For each mention-entity pair (mi, ei) in a batch of B pairs, the training loss is computed as follows:
LE = ∑
X∈{∅,T,V,C} − log( exp(SX(mi, ei))∑B j exp(SX(mi, ej)) )
(7) where ei is the gold entity of the mention mi. Eventually, the loss of our MELOV is calculated as:
L = LE + λLG (8)
where λ is the hyperparameter to control the loss.

heading: 4 Experiments

heading: 4.1 Datasets and Baselines
text: We evaluate our MELOV under three public multimodal entity linking datasets: WikiMEL (Wang et al., 2022a), RichpediaMEL (Wang et al., 2022a), and WikiDiverse (Wang et al., 2022b). Table 1 shows the overall statistics of these three datasets. For fair comparison, following previous works (Wang et al., 2022a; Luo et al., 2023), we utilize a subset KB of Wikidata as our entity set in each dataset. For WikiMEL and RichpediaMEL, the
data is split into 70% training, 10% validation and 20% test sets respectively. As for WikiDiverse, the proportions are 80%, 10% and 10%.
For the quantitative evaluation of our proposed MELOV, we utilize the following competitive methods for comparison. The first type of baselines is the text-based methods that only use the textual modality information, including BLINK (Wu et al., 2020), and BERT (Devlin et al., 2019). Another type is the language-and-vision pre-training models, including CLIP (Radford et al., 2021), ViLT (Kim et al., 2021), and METER (Dou et al., 2022). Furthermore, our baselines also contain state-ofthe-art MEL methods, including DZMNED (Moon et al., 2018), JMEL (Adjali et al., 2020), VELML (Zheng et al., 2022), GHMFC (Wang et al., 2022a), and MIMIC (Luo et al., 2023).

heading: 4.2 Implementation Details
text: In our experiments, following (Luo et al., 2023), we initial the weights of BERT and ViT by using the pre-trained CLIP-Vit-Base-Patch32 version. The maximum sequence length of words for textual input is set to 40. All images are rescaled into 224×224 resolution and the patch size P is 32. We use the AdamW (Loshchilov and Hutter, 2019) optimizer with a batch size of 128 for optimizing. We train our MELOV 20 epochs with a learning rate of 1e-5. Experiments were conducted on a PC with 256 GB RAM, 4 Intel(R) Xeon(R) Gold 6226R CPUs and an NVIDIA GeForce RTX A6000 GPU with 48 GB memory. For evaluation, we utilize the H@k and MRR as our metrics. H@k represents the hit rate of the corresponding entity among the top-k ranked entities. And MRR indicates the mean
30 40 50 60 70 80 90 100
A
reciprocal rank of the corresponding entity.

heading: 4.3 Overall Performance
text: Table 2 shows the experimental results of our MELOV in comparison with baselines on three MEL benchmark datasets. We have the following observations. Firstly, the text-based methods BLINK and BERT perform unsatisfactorily since they only rely on textual inputs and ignore visual information. Secondly, some multimodal methods, such as ViLT and JMEL, exhibit inferior performance compared to text-based methods. This observation indicates that shallow modality interaction and naive multimodal fusion do not enhance, and may even degrade, the performance of MEL. Thirdly, most multimodal methods, such as CLIP, GHMFC and MIMIC, perform significantly better than text-based methods. And MIMIC achieves the best results among all baselines, which demonstrates the effectiveness of integrating visual information and the superiority of our chosen multigrained interaction in our multimodal entity linking module. Finally, our MELOV outperforms all baselines on all three datasets and achieves new state-ofthe-art performance. Specifically, MELOV gains 0.93%, 3.12% and 3.81% absolute improvement of H@1 on WikiMEL, RichpediaMEL and WikiDiverse respectively. This suggests the effectiveness of optimizing visual features in latent space and the superiority of our MELOV.

heading: 4.4 Ablation Study
text: To better understand our proposed MELOV, we conduct a series of ablation studies, as also presented in Table 2. We can observe that MELOV w/o inter-modality generation and MELOV w/o intra-modality aggregation both result in a de-
crease in performance, confirming the effectiveness of these two optimizations. The performance of MELOV w/o inter drops more on WikiMEL, while MELOV w/o intra drops more on RichpediaMEL and WikiDiverse. This is reasonable since different datasets have distinct optimization requirements. We also find that our MELOV outperforms MELOV w/o τ , highlighting the importance of utilizing such a learnable threshold in the similarity matrix to avoid the noise brought by dissimilar mentions. Finally, we replace the whole optimization process with the filtering way proposed by Zhang et al. (2021). We observe that this leads to a significant performance drop and MELOV (filter) even performs worse than MIMIC on RichpediaMEL and WikiDiverse. This is consistent with our claim that many poor images still possess valuable implicit visual cues. Discarding them outright would result in the loss of significant complementary visual information, ultimately causing a significant degradation in performance. And our method, optimizing visual features in latent space, is a better way to handle poor images.

heading: 4.5 Hyperparameter Analysis
text: To investigate the impact of hyperparameters on the performance of our MELOV, we vary them (the latent dimension size dz , number of GCN layers L, and final loss hyperparameter λ) on RichpediaMEL and WikiDiverse, which is shown in Figure 3. Firstly, we find that dz has a significant impact on the final performance by influencing the effect of generating visual features. A small dimension may make our model inadequate to mine shared information from heterogeneous textual features, while a large one may cause redundancy to lead to performance degradation. Secondly, increasing the number of GCN layers does not always lead to improved performance. Excessive message passing and aggregation can potentially exacerbate the issue of data sparsity. Finally, λ, which is used to control the strength of LG, also affects the performance. In general, our MELOV obtains best results while setting dz as 64, L as 2, and λ as 0.001.

heading: 4.6 Analysis of Similarity Calculation
text: To explore the effect of employing various similarity metrics in computing the GCN similarity matrix on our MELOV performance, we compare our kernel-driven approach with three traditional similarity metrics (i.e. dot product similarity, Cosine similarity and Euclidean distance), as presented in Table 3. Upon observation, Euclidean distance outperforms the other two traditional metrics, indicating its efficacy in capturing numerical disparities among feature nodes. Furthermore, kernel-driven approach demonstrates superior performance in comparison to these traditional metrics, highlighting the effectiveness of our chosen similarity calculation method. This approach can adequately capture local manifold structure and complex relationships such as higher-order statistics.

heading: 4.7 Case Study
text: To conduct a qualitative analysis, Table 4 presents three WikiDiverse cases comparing our MELOV with the current state-of-the-art baseline MIMIC. Notably, all the three cases contain poor mention images. They will mislead MEL models into linking to wrong text-related entities, armed forces related entities and human-related entities respectively, like MIMIC. However, our MELOV effectively leverages the shared information from heterogeneous textual features and relevant visual details of semantic similar neighbors to optimize visual features in latent space to avoid this phenomenon, which helps to make correct decisions.

heading: 4.8 Performance on Zero-Shot Setting
text: With the proliferation of vast amounts of data on the web, new entities are emerging constantly. As shown in the left part of Figure 5, all three datasets contain a large number of unseen samples, whose entities never appear during training. Thus, it is
necessary to investigate the performance of MEL models on the zero-shot setting, which mainly focuses on unseen entities. We conduct experiments on these unseen samples and the results are shown in the right part of Figure 5. We can find that our MELOV significantly outperforms the existing state-of-the-art model MIMIC on all three datasets. Through the joint optimization of intermodality generation and intra-modality aggregation, our MELOV can learn richer and more suitable visual representations, thereby enhancing the generalization ability on new entities.

heading: 5 Conclusion
text: In this paper, we focus on poor images in multimodal entity linking. To avoid the negative effects of poor images while preserving the implicit visual cues of original images, we propose MELOV, a novel joint optimization framework to combine inter-modality generation and intra-modality aggregation to optimize visual features in latent space. This effectively leverages the shared information from heterogeneous textual features and relevant visual details of semantic similar neighbors, allowing for simultaneously handling all types of
poor images. Experimental results on three public multimodal entity linking datasets demonstrate the effectiveness of optimizing visual features in latent space and our proposed MELOV achieves new state-of-the-art performance.
Limitations
Although our MELOV has demonstrated its effectiveness on the multimodal entity linking task, there are still some limitations to be addressed in the future: 1) In this work, for inter-modality generation, we only utilize the variational autoencoder to mine shared semantic information from heterogeneous textual features and generate vision-specific features. However, other state-of-the-art generation frameworks also can be used and may work better. We leave the exploration of other generation frameworks to future work. 2) Another limitation is the resource limitation. We argue that the more resources we leverage, the richer and more suitable the optimized visual features will be. However, for a fair comparison with baselines, we only use the inter-modality heterogeneous textual information and intra-modality related visual information. We will investigate to utilize more resources to optimize visual features in future work.

heading: Acknowledgements
text: This research is supported by the National Natural Science Foundation of China (No. 62272250, 62302243, U22B2048), the Natural Science Foundation of Tianjin, China (No. 22JCJQJC00150, 23JCYBJC01230), and the Fundamental Research Funds for the Central Universities, Nankai University (63241442).

heading: Future Work
text: Limitations
Although our MELOV has demonstrated its effectiveness on the multimodal entity linking task, there are still some limitations to be addressed in the future: 1) In this work, for inter-modality generation, we only utilize the variational autoencoder to mine shared semantic information from heterogeneous textual features and generate vision-specific features. However, other state-of-the-art generation frameworks also can be used and may work better. We leave the exploration of other generation frameworks to future work. 2) Another limitation is the resource limitation. We argue that the more resources we leverage, the richer and more suitable the optimized visual features will be. However, for a fair comparison with baselines, we only use the inter-modality heterogeneous textual information and intra-modality related visual information. We will investigate to utilize more resources to optimize visual features in future work