Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking (2024)

Abstract: Multimodal Entity Linking (MEL) aims to link ambiguous mentions in multimodal contexts to entities in a multimodal knowledge graph. A pivotal challenge is to fully leverage multielement correlations between mentions and entities to bridge modality gap and enable finegrained semantic matching. Existing methods attempt several local correlative mechanisms, relying heavily on the automatically learned attention weights, which may over-concentrate on partial correlations. To mitigate this issue, we formulate the correlation assignment problem as an optimal transport (OT) problem, and propose a novel MEL framework, namely OTMEL, with OT-guided correlation assignment. Thereby, we exploit the correlation between multimodal features to enhance multimodal fusion, and the correlation between mentions and entities to enhance fine-grained matching. To accelerate model prediction, we further leverage knowledge distillation to transfer OT assignment knowledge to attention mechanism. Experimental results show that our model significantly outperforms previous state-of-the-art baselines and confirm the effectiveness of the OT-guided correlation assignment.1

heading: 1 Introduction
text: Entity Linking (EL) is an important yet challenging task in knowledge acquisition, which can facilitate applications such as information retrieval (Liu et al., 2018), question answering (Xiong et al., 2019) and dialogue systems (Jiang et al., 2023). In recent years, with the rapid increase of multimodal content, Multimodal Entity Linking (MEL) has been proposed, which aims to link ambiguous mentions in multimodal contexts to entities in the multimodal knowledge graph (KG). As shown in
* These authors contribute equally to this work. † Corresponding Authors. 1Source code of this paper could be obtained from
https://github.com/zhangzef/OT-MEL
Figure 1, given a mention Robert in texts with related images, the MEL model requires the mention to be corresponded to the entity Robert_Downey in the multimodal KG.
Since the mention can be ambiguous with limited textual information, it is crucial to enrich the text with its related visual images. To utilize multimodal information for MEL, a pivotal challenge is to leverage multi-element correlations within entity and mention contexts: 1) the multi-element correlations between modalities, where the textual tokens and visual patches may have correlative correspondence. As shown in Figure 1, the token Robert and Holland correspond to different local areas of human faces in the image. These correlations serve as message-passing pivots between the text and image, which can be valuable to bridge the semantic gap of modalities (Luo et al., 2023). 2) the multi-element correlations between mentions and entities, where the context of mentions and entities can have multiple correlative cues for semantic matching. As shown in Figure 1, the mentions and entity both involve token Robert, and both the mention and the entity involve the word Actor. These correlations depict similarities between the mention and entity, providing explicit evidence for EL.
4103
To facilitate these multi-element correlations, existing studies propose sophisticated correlative mechanisms, such as feature concatenation (Adjali et al., 2020), attention-based modules (Moon et al., 2018; Wang et al., 2022a), hierarchical gates (Luo et al., 2023; Wang et al., 2022a) and graph-based aggregation (Xing et al., 2023). However, these models learn the correlations of elements locally, and rely heavily on the automatically learned correlation assignments. This design may over-concentrate on part of elements, dominating the correlations without global consideration. For example, all entities in Figure 1 have surface name Robert the same to mention Robert, which may attract the correlation assignment and conceal crucial correlations of token Actor in texts and the visual patch of human face in images.
To mitigate the above issues, we formulate the multi-element correlation assignment as an optimal transport (OT) problem. Taking the multimodal correlation as an example, given the cost of each pair of textual tokens and visual patches, the OT problem aims to find an optimal transport plan to transfer the token to a patch with assigned weights, such that the total cost is minimal. In contrast to the automatically learned attention map, the OT plan globally considers the correlation between source and target elements, surpassing the degraded dominant correspondence (Liu et al., 2020a).
Following the above idea, we propose a novel MEL framework with OT guided correlation assignment, termed OT-MEL. Specifically, we first adopt multimodal feature encoders to represent textual tokens and visual patches for both mentions and entities. Afterward, we propose an OT-based correlation assignment method between textual tokens and visual patches, with which we further integrate multimodal features and bridge the semantic gap of modalities. Additionally, we also build unimodal correlations between mentions and entities with OT-based correlation assignment. By measuring the similarity of the multimodal and unimodal features between mentions and entities, the model derives the overall matching score. To enhance efficiency, we further propose distillation between OT assignment to attention map, such that we can use attention to approximate OT plan. The contributions of this paper are three-fold:
• We introduce optimal transport to capture complicated correlations in MEL. To our knowledge, we are the first to guide MEL with OT.
• We propose OT-MEL, a novel framework to consider the multimodal and unimodal correlations, which can be further extended by knowledge distillation from OT plan to attention mechanism.
• Empirical studies indicate that our model outperforms previous state-of-the-art methods on three widely-used benchmarks and extensive analysis confirm the effectiveness of our model.

heading: 2 Preliminary

heading: 2.1 Task Formulation
text: The task of MEL aims to correspond a given mention from text within a multimodal context to its true entity in a multimodal KG. Generally, the KG involves entities e ∈ E , where each entity e ≜ {en, ev, ea} can be associated with entity name en, visual image ev, and a textual attribute set ea 2. Besides, the given mention from texts can be summarized as m ≜ {mn,mc,mv}, which is associated with mention word mn, context sentence mc, and visual image mv, respectively.
To achieve this task, for each given mention, the model requires to identify the true entity from entity candidates, which can be formulated as:
θ∗ = argmax θ
∑
(m,e)∈D log pθ(e|m, E), (1)
where θ∗ denotes the final model. Following previous studies (Luo et al., 2023), we use all entities in KG as the candidates.

heading: 2.2 Optimal Transport
text: Optimal Transport (OT) aims to find a transport plan with minimal total cost, which transports the density distribution of a group of elements to that of another group, given the cost of each element pair (Villani et al., 2009). Formally, given element xi in one group, yj in another group and Cij the cost between them, the problem can be formulated as the following linear programming problem:
Π⋆ = argmin Π∈Rn×m+
m∑
i=1
n∑
j=1
CijΠij ,
s.t.Π1n = µ, Π ⊤1m = ν,
(2)
where Π∗ is the optimal transport plan, where Πij is the amount of information from xi to yj . 1n
2Following most existing MEL studies(Wang et al., 2022a; Luo et al., 2023), we also treat the attributes as textual information of the entities due to the large-scale KG consideration.
and 1m are all-ones vectors of dimension n and m, respectively. In this way, the transport plan remains the multi-element correlations between two groups with appropriate weight assignments.
To accelerate the calculation, Cuturi (2013) proposes to smooth the problem with the entropy regularization term as follows:
Π⋆ = argmin Π∈Rn×m+
m∑
i=1
n∑
j=1
CijΠij − λH(Π),
s.t.Π1n = µ, Π ⊤1m = ν,
(3)
where H(Π) is the entropy of Π, λ is regularization coefficient. Based upon, the OT problem can be efficiently solved by Sinkhorn-Knopp algorithm, detailed in Algorithm 1.

heading: 3 Method
text: In this section, we introduce our OT-MEL model, which consists of three modules: 1) multimodal feature encoder, 2) multimodal feature interaction, and 3) multimodal entity linking. The architecture of OT-MEL is shown in Figure 2.

heading: 3.1 Multimodal Feature Extraction
text: Given the mention m and entity e with their text and image, we are going to generate their initial embeddings. To enrich fundamental correlations between multimodal features, following Luo et al. (2023), we adopt contrastive language-image pretraining architecture (CLIP) (Radford et al., 2021) as the multimodal encoder, which contains a pretrained BERT (Devlin et al., 2018) for textual embedding and a pre-trained ViT (Dosovitskiy et al.,
2020) for visual embedding. For entities and mentions we use a parameter shared encoder to ensure consistency in the representation space.
Textual Feature Extraction We employ BERT to extract textual features. For mention m, we concatenate the word of mention mn with its context sentence mc, and then feed it into BERT as:
Tm = BERT([CLS]mn[SEP ]mc[SEP ]), (4)
where Tm ∈ RLmt×d is the mention textual embeddings. On the other hand, for entity e, we concatenate the entity name en with its attributes3 ea as [CLS]en[SEP ]ea[SEP ], and use the same way to obtain the entity textual embeddings Te ∈ RLet×d. Visual Feature Extraction We employ the pretrained ViT to extract visual features. To process the mention image into token sequence, we reshape the image mv ∈ RH×W×C into a sequence of flattened 2D patches as mvp ∈ RP×P×C with special token [CLS], and feed them into ViT as:
Vm = ViT([m v [CLS],m v 1, ...,m v lv−1]), (5)
where Tm ∈ RLmv×d is the mention visual embeddings, and lv = HW/P 2 is the resulted patch number. Similarly, we reshape the image of entity into patches with the same patch size, and obtain the entity visual embeddings Ve ∈ RLev×d.

heading: 3.2 Multimodal Feature Interaction
text: As claimed in the introduction, the text and image can have multi-element correlations served as
3Following Luo et al. (2023), we also treat different attributes as texts separated by period.
message passing pivots. Therefore, we assign distinct weights for multimodal feature interaction. Take the mention-side texts and images as examples, we first introduce an intuitive attention-based assignment, and thereupon propose our OT-based correlation assignment method.
Attention-based Correlation Assignment To interact multimodal features with distinct weights, an idea is to employ attention mechanism (Vaswani et al., 2017) between the text and image. Given the mention m with its textual embeddings Tm and visual embeddings Vm, we establish the image-totext correlation assignments as follows:
Q = TmWq,K = VmWk,H = VmWv, (6) Av→t = softmax(S),S = QK⊤/ √ d (7) Gv→tm = A v→tH (8)
where Wq, Wk, Wv are learnable parameters with Rd×d. Here Av→tij ∈ RLmt×Lmv is the assigned correlation weight from visual patch Vm,j to textual token Tm,i. As claimed, the attention-based assignment focuses on local similarity of elementpairs without global consideration.
OT-based Correlation Assignment To globally consider the correlations, we propose OT-based correlation assignment method. Practically, we formulate the image-to-text correlation assignment as an OT problem, where we hope the textual tokens and visual patches has appropriate correlation with minimal total transport cost. Therefore, we project original textual feature Tm and visual feature Vm into an assignment feature space, and define the transport cost from patches to tokens with a measurement ϕ as follows:
Q = TmWq,K = VmWk,H = VmWh, (9)
Cij = ϕ(Qi,Kj) ≜ 1
2 [1− cos(Qi,Kj)], (10)
where we achieve ϕwith cosine similarity such that ϕ ∈ [0, 1]. In this way, the token-patch pairs with higher similarity would has lower transport cost. Based upon, we treat each element of tokens and patches equally as uniform distributions, and solve the correlation assignment problem by SinkhornKnopp algorithm (Cuturi, 2013):
Av→t = Π⋆=Sinkhorn-Knopp(C,µ,ν), (11) Gv→tm = A v→tH (12)
where µ = 1/Lmt1mt and ν = 1/Lmv1mv are the probabilistic simplexes of textual tokens and visual patches, respectively. In this way, we establish the image-to-text correlations with OT assignment in Eq. (11), which ensures the overall transport cost to be minimal. Actually, this design treats all tokens and patches equally, and ensures all token-patch pairs to be appropriately transported according to the similarity-related cost, thus avoiding the suboptimal assignment dominated by partial elements. In summary, the above procedure holds for both mentions and entities, and all generated features are termed as Gv→tm , G t→v m , G t→v e and G v→t e .

heading: 3.3 Multimodal Entity Linking
text: With the interacted multimodal features, we conduct MEL by matching the mentions and entities. To enhance the interaction between unimodal features of mentions and entities, we also conduct OT-based correlation assignment on them.
Multimodal Feature Matching To fully consider the multimodal information, we integrate the interacted features to match mentions and entities, respectively. Take mentionm as an example, we integrate the image-to-text representation with its textual embedding Tm and the transported embedding Gv→tm from its image. Generally, we can employ any pooling method to aggregate these embeddings as G = {Tm,Gv→tm }. Here we introduce a simple yet effective method, softpool (Stergiou et al., 2021), which is a soft version of max-pooling:
ḡtm = softpool(G) ≜ ∑
gi∈G wi ⊙ gi, (13)
wi = exp(gi)∑
gj∈G exp(gj) (14)
where ḡtm ∈ Rd is the generated textual representation, ⊙ is Hadamard production. In this way, it emphasizes the most activated features in the generated multimodal representation.
Based upon, we obtain the text-to-image representations of mention m, and those representations of entity e as follows:
ḡvm = softpool({Vm,Gt→vm }) (15) ḡte = softpool({Te,Gv→te }) (16) ḡve = softpool({Ve,Gt→ve }) (17)
Finally, we measure the matching scores between mentions and entities, considering both of their
textual and visual representations:
SF(m, e) = ψ(ḡ t m ⊕ ḡvm, ḡte ⊕ ḡve ) (18)
where ψ is the similarity measurement, and we achieve it by vector inner production. SF(m, e) is the fused multimodal feature matching score.
Unimodal Feature Matching Considering the unimodal correlations between mentions and entities, we design unimodal feature matching module with OT-based correlation assignment. Specifically, take the text-modal as an example, we capture the correlations between mention textual feature Tm to entity Te. We also derive the cost matrix with cosine similarity between them, and obtain the textual modal correlated mention representation:
Q = TeW ′ q ,K = TmW ′ k,H = TmW ′ v, (19) Cij = ϕ(Qi,Kj) ≜ 1
2 [1− cos(Qi,Kj)], (20)
Am→e=Π⋆= Sinkhorn-Knopp(C,µ,ν), (21) Gm→et = A m→eH (22)
where µ = 1/Lmt1mt and ν = 1/Let1et are the probabilistic simplexes. Based upon, we aggregate the textual representations for the mention, and achieve the unimodal feature matching between mention and entity by:
ḡm = softpool({Gm→et }) (23) ST(m, e) = 1
2 [ψ(ḡm, te) + ψ(tm, te)] (24)
where we also use inner production to achieve ψ. Here, tm and te are the output embedding of token [CLS], remaining the original summarized feature of texts. This precedure also holds for visual modal feature matching with matching score SV(m, e).
Overall Matching Score To consider different aspects in feature matching, we collect all matching scores and obtain the overall matching score as:
SO(m, e) = 1
3
∑
X∈{F,T,V} SX(m, e) (25)
Thereby, we design contrastive training objective on the overall matching score as follows:
LO = − log exp(SO(m, e))∑
e′∈E exp(SO(m, e ′)) , (26)
where E is all candidate entities in the KG. For training, we adopt all other entity e′ except the true
entity e in the batch, which serve as the in-batch negative samples. To enhance the consistency in different matching aspects, following Luo et al. (2023), we also apply contrastive training objective on different matching scores:
LX = − log exp(SX(m, e))∑
e′∈E exp(SX(m, e ′)) ,
L = LO + ∑
X∈{F,T,V} LX
(27)
where X ∈ {F,T,V} is the original multimodal and unimodal matching scores. L is the overall loss of our model with the full OT-based version.

heading: 3.4 Optimal Transport Distillation
text: To improve the efficiency of OT-based correlation assignments, we further propose an efficient version with KD, which transfers knowledge from OT-based correlation assignment to attention-based correlation assignment. Generally, we can use mean squared error (MSE) loss to assimilate the assignment Av→t in Eq. (7) and Eq. (11). To better consider both the source and target aspects in OT assignment, we design Kullback-Leibler (KL) divergence to fully transfer knowledge as follows:
KD(Av→tOT ,A v→t ATT)
≜ 1 2 [ ∑ jKL(softmax(Π ⋆ i∗), softmax(Si∗)) + ∑
jKL(softmax(Π ⋆ ∗j), softmax(S∗j))]
(28) Here, we conduct knowledge distillation on all OTbased correlation assignments, including Am,v→t, Am,t→v,Ae,v→t, Ae,t→v for multimodal feature interaction, and Am→e, Ae→m for unimodal feature matching. All these knowledge distillation constraints constitute the overall KD loss LKD. Alternatively, the overall training loss is:
J = L+ LKD (29)
Here, J is the overall loss with knowledge distillation, which obtains an efficient KD-version model.

heading: 4 Experiments
text: In this section, we carried out comprehensive experiments on three public MEL benchmark datasets to evaluate the effectiveness of OT-MEL model.

heading: 4.1 Datasets
text: In the experiments, we selected three public MEL datasets RichpediaMEL with 17K samples,
WikiMEL (Wang et al., 2022a) with 25K samples and WikiDiverse (Wang et al., 2022b) with 15K samples to verify the effectiveness of our proposed method. We follow the training, validation, and testing set splitting approach as Luo et al. (2023). Following Wang et al. (2022a) we use the subset of Wikidata as KG for each dataset.The detail of three datasets information are reported in Appendix C.

heading: 4.2 Baselines
text: We compare our method with various competitive baselines in three groups: 1) the text-based methods, which utilize textual information only to achieve EL, including BLINK (Wu et al., 2019), BERT (Devlin et al., 2018), and RoBERTa (Liu et al., 2019). 2) the vision-and-language pretraining (VLP) methods including CLIP (Radford et al., 2021), ViLT (Kim et al., 2021), ALBEF (Li et al., 2021), and METER (Dou et al., 2022). 3) the MEL enhanced methods4, which explore sophisticated interaction networks to capture diverse correlations, including DZMNED (Moon et al., 2018), JMEL (Adjali et al., 2020), VELML (Zheng et al., 2022), GHMFC (Wang et al., 2022a), MIMIC (Luo et al., 2023). For details, we provide extensive descriptions of baselines in Appendix B.

heading: 4.3 Implementation Details
text: Our model weights are initialized with pre-trained CLIP-Vit-Base-Patch32. We set the batch size to
4For fairness, we don’t compare with DRIN (Xing et al., 2023), since we fail to run the model due to the numerous scene graphs of entities.
256, the learning rate to 2× 10−5, and the hidden dimension d to 96. For the OT parameter λ, we set it to 0.6. All methods are evaluated on the validation set and the checkpoint with the highest MRR is selected to evaluate on the test set.

heading: 4.4 Main Results
text: We conduct comparative experiments on three public datasets to verify the effectiveness of each model on MEL tasks, Table 1 shows the H@k and MRR of different models in the three datasets, and we will provide the formula for metrics in the Appendix D. Based on the experimental results, we further have the following observations and analysis: (1) Multimodal information is important for EL problems. The first section of the table presents text-based EL models, which, despite demonstrating commendable performance, significantly lag behind the multimodal models showcased in the second section of the table. (2) Our model is effective on all three datasets. Compared with the state-of-the-art model, OT-MEL improves H@1 by 2.28%, 2.58% and 0.99% on the three datasets, respectively. (3) OT can indeed guide the model for better modality fusion. The results of OT-MEL(KD) based on distillation on 3 datasets also exceed the current state-of-the-art models.

heading: 4.5 Variant Analysis
text: To inspect the effectiveness of the model components, we show results of experiments for model variants in Table 2, where we find: (1) OT is more effective than attention mechanisms in both uni-
modal and multimodal feature matching. In models F1 and M1, where we replaced OT with attention, there was a noticeable decrease in performance for both. (2) Multimodal feature fusion can better utilize the complementary information between modalities to aid in linking. In model F2, after removing the multimodal feature fusion module, the average results decreased by 3.61%. (3) Unimodal feature matching can more effectively utilize the fine-grained correlations between images and texts to enhance EL. In model M2, when the unimodal feature matching module is removed, the performance decreases by 7.89%. (4) Compared to mean pooling and max pooling, SoftPool can aggregate sequential data more effectively. After replacing SoftPool with mean pooling and max pooling, the performance decreased by 2.11% and 1.02%, respectively. (5) OT-MEL(KL) can enhance the inference speed of the model while only marginally compromising model performance. OT-MEL(KL), compared to OT-MEL(OT), improves the inference speed by 337.20s on the validation set of the Richpediamel dataset, while the performance only decreases by 0.62%. Table 4 displays the average inference speeds on the validation sets of different datasets.

heading: 4.6 Analysis on MEL with Multi-objects
text: To validate that our model can better utilize the multi-element correlations between different modalities to aid in EL, we used YOLOv5 for object detection on images in the Richpediamel test dataset. We categorized images into multi-object sets for object counts of two or more, and single-
object sets for counts of one or fewer. Notably, we excluded data where EL was straightforward due to identical mentions and entity names with no ambiguous counterparts in the KG, ensuring test rigor. Table 3 shows the results of our experiment. Observations indicate that our model can indeed achieve better linking by utilizing the multi-element correlations between multimodalities. Compared to the attention mechanism, our model shows a 6.19% improvement in H@1 on single-object sets and a 10.40% improvement in H@1 on multi-object sets.

heading: 4.7 Case Study of Optimal Transport
text: The entropy controls the information dispersion in the transport plan of OT, where the higher entropy tends to have a more disperse concentration. Figure 3 shows H@1 and MRR results. The observation reveals that an appropriate entropy allows the model to better focus on the information it should. We can see that when the entropy is low, the results are poor due to an overly concentrated focus. Besides, when the entropy is high, the results tend to be worse due to an overly concentrated focus.

heading: 5 Related Works
text: Text-based Entity Linking Traditional EL mainly focuses on the textual information of mentions and entities. These methods (Cao et al., 2017; Wu et al., 2019) first employ textual encoders to
obtain contextual representations of mention and entity, and then calculate similarities between them as final probabilities. To enhance consistency in EL, they enrich local correlations (Peters et al., 2019; Liu et al., 2023) with prior knowledge and collective mechanism, and global correlations (Le and Titov, 2018; Cao et al., 2018; Fang et al., 2019) with topical coherence among mentions. However, these methods are designed to deal with text only, which can hardly deal with the crucial multimodal correlations within images in MEL. Other studies have focused on Generative Entity Linking (GEL) (De Cao et al., 2020; Shi et al., 2023; De Cao et al., 2022; Huang et al., 2022; Yuan et al., 2022). This is a promising research problem in the era of Large Language Models (LLMs), but they can suffer from the lexical ambiguity problem (Fortuny and Payrató, 2024), which we will discuss in details in the Appendix G.
Multimodal Entity Linking Recent years have seen numerous social media and news posted with texts and images, thereby MEL is proposed to identify interested entities from multimodal information, which can benefit to knowledge acquisition (Sheng et al., 2020; Li et al., 2023). As a pioneering study, Moon et al. (2018) firstly introduces images to benefit EL, which blends visual, word and character features through an attention mechanism. Afterward, there exist several studies (Adjali et al., 2020; Zheng et al., 2022; Wang et al., 2022a; Luo et al., 2023) exploring gate or attention mechanism to capture complicated multimodal correlations for mentions and entities. Recently, Xing et al. (2023) introduce external object-level scene graphs of images to enrich visual correlations, but may
involve elaborate errors and still construct multimodal correlations with attentive graph neural networks. Luo et al. (2023) present a sophisticated gated framework for fine-grained inter-modal interactions, achieving the most advanced results. Although existing studies attempt multiple correlations in different aspects, they heavily rely on the automatically learned attention-based mechanism, which can hardly ensure appropriate correlation assignments and impede further improvements.
Optimal Transport OT (Kantorovich, 1942) is a well-known problem, which can trace back to Monge (1781). The key idea is to derive an optimal transport plan to transfer one distributions to another (Arjovsky et al., 2017; Chen et al., 2020; Bhardwaj et al., 2021; Yang et al., 2023). There are extensive studies utilizing the transport plan of OT to solve assignment problems (Chang et al., 2022; Cao et al., 2022). Liu et al. (2020b) employ the global OT plan to address the many-to-one issue in semantic correspondence in computer vision. Lee et al. (2019); Chen et al. (2020) employ the wasserstein distance to align the representation of text and image. However, few existing studies conduct comprehensive investigation of the OT plan in the context of multi-element correlations within both unimodal and multimodal settings.

heading: 6 Conclusion
text: This paper introduces a MEL model with OTguided correlation assignment, termed OT-MEL. Previous studies rely heavily on the automatically learned attention weights, which may overconcentrate on partial correlations. To this end, OTMEL proposes capturing these multi-element correlations across modalities with OT. To improve the efficiency, we further transfer knowledge from OT to attention mechanism. Experiments on 3 datasets illustrate that our model outperforms current stateof-the-art models. In the future, we will explore other efficient method to deploy OT in MEL.
Limitations
In this paper, we propose a MEL model with OT correlation assignment, namely OT-MEL. First, OTMEL simplistically assumes uniform distributions for both source and target, which can be further explored in real-world scenarios. Besides, as OTMEL is tailored for visual and textual modality in MEL, it doesn’t consider OT in other modalities such as speech and video. We will explore the generality of the proposed OT correlation assignment in these complex modalities for wide research communities.

heading: Acknowledgement
text: We would like to thank the anonymous reviewers for their comments. This work was supported by the National Key Research and Development Program of China (Grant No.2021YFB3100600), the Youth Innovation Promotion Association of CAS (No.2021153), and the Postdoctoral Fellowship Program of CPSF (No.GZC20232968).

heading: A Optimal Transport
text: Optimal Transport (OT) is a problem which traces back to the work of Gaspard Monge (Monge, 1781). It is a constrained optimization problem that seeks an efficient solution of transporting one distribution of mass to another. Let µ and ν be two probability measures defined on the same space Rn, representing the source and target distributions of resources, respectively. The Monge problem aims to find a mapping Π : Rn → Rn that transforms a distribution µ into a distribution ν while minimizing the total transfer cost. This cost is usually measured by a cost function c(x,Π(x)), where x is the original location of the resource and Π(x) is the new location to which it has been transferred. It can be formulated as the following optimization problem:
Π∗ = argmin Π
∫
Rn c(x,Π(x)) dµ(x), (30)
where Π is the transport map, c(x,Π(x)) is a cost function, measure from the x to the Π(x) transfer cost, and the integral is the total cost over all x.
However, there is a very strong assumption in the Monge problem which is the inseparability of masses, that is, the Monge problem requires an explicit mapping Π to transfer directly from one location to another, and when the two distributions have different total masses or there is a many-to-many mapping between the source distribution and the target distribution, the Monge problem cannot find a feasible solution. For example, in multimodal problem, the total mass in the image distribution and the total mass in the text distribution may be different, and an image patch may correspond to multiple text tokens. Kantorovich (Kantorovich, 1942) proposed an extension and generalization of the Monge problem to address the aforementioned limitations in the Monge problem. Figure 5
illustrates the difference between the Monge Formulation and the Kantorovich Formulation.
Algorithm 1: Sinkhorn-Knopp Algorithm Require: Cost matrix C ∈ Rn×m, marginals µ ∈ Rn, ν ∈ Rm, regularization parameter λ > 0
Ensure: Approximate transport matrix P Initialize u← 1n, v← 1m Compute K ← exp(−λC) while not converged do
Update u← µ⊘ (Kv) Update v← ν ⊘ (KTu)
end while Compute P← diag(u)Kdiag(v) return P
Instead of finding a specific transport map, the Kantorovich problem looks for a transport plan, which is a more flexible notion. In this problem, we consider two given probability distributions µ and ν, defined on two spacesX and Y , respectively. The Kantorovich problem aims to find a joint probability distribution (transport plan) Π over X × Y whose marginal distributions are µ and ν, respectively, while minimizing the overall transport cost. Mathematically, this problem can be expressed as follows:
Π⋆ = argmin Π
∫
X×Y c(x, y)Π(x, y) dx dy,
s.t. ∫
Y Π(x, y)dy = µ,
∫
X Π(x, y)dx = ν,
(31) where Π is a joint probability measure on X × Y , representing the transition plan, and c(x, y) is a cost function for moving from x ∈ X to y ∈ Y .
Within the scope of our problem, similar to the previous optimal transport (OT) settings (Arase et al., 2023; Chang et al., 2022), we simply assume the tokens of the text and the patches of the image as two different uniform distributions, which means that each token or patch has the same amount of information. Each token or patch is a sample from a discrete probability distribution. So we need to use the Kantorovich problem in discrete form to compute the optimal transport plan between text and image. Specifically, let X = {x1, x2, . . . , xn} be the token set of text and Y = {y1, y2, . . . , ym} be the patch set of image. The discrete form of Kantorovich problem can be formulated as finding a "transport plan" Π, which is a matrix n×m,
where each element Πij represents the amount of information to be transported from text token xi to image patch yj . This plan needs to minimize the total transportation cost while satisfying constraints on the supply and demand of information. Mathematically, this problem can be formulated as the following linear programming problem:
Π⋆ = argmin Π∈Rn×m+
m∑
i=1
n∑
j=1
cijΠij ,
s.t. Π1n = µ, Π⊤1m = ν,
(32)
where Π∗ is the optimal transport plan, Πij is the amount of information from xi to yj in the optimal transport plan, Rn×m+ denotes the space of n×m real matrices with all non-negative elements, cij denotes the unit transport cost from xi to yj . 1n is an all-ones column vector of dimension n, and 1m is an all-ones column vector of dimension m. Since the computation of the Kantorovich problem involves the resolution of a linear program, its cost is prohibitive when dealing with large-scale data. Therefore, Cuturi (2013) proposes to smooth the kantorovich problem with the entropy regularization term:
Π⋆ = argmin Π∈Rn×m+
m∑
i=1
n∑
j=1
cijΠij − 1
λ H(Π),
s.t. Π1n = µ, Π⊤1m = ν,
(33)
where H(Π) is the entropy of Π, λ > 0 is regularization coefficient, then solve the optimal transport problem quickly by Sinkhorn-Knopp algorithm. Algorithm 1 shows the flow of Sinkhorn-Knopp algorithm, where ⊘ is element-wise division.

heading: B Description of Baselines
text: We compared our method with various competitive baselines. 1) The text-based methods include:
• BLINK (Wu et al., 2019) a two-stage zero-shot EL method, utilizes BERT as its backbone. It first retrieves entities using a bi-encoder, then re-ranks them with a cross-encoder.
• BERT (Devlin et al., 2018) consists of a stack of Transformer encoders and is pre-trained on a large amount of corpus.
• RoBERTa (Liu et al., 2019) further improves BERT by removing next sentence prediction objective and using a dynamic mask strategy.
2) The Vision-and-Language Pre-training (VLP) methods include:
• CLIP (Radford et al., 2021) uses two Transformer-based encoders for visual and textual representation, pre-trained on massive noisy web data using contrastive loss.
• ViLT (Kim et al., 2021) adopts shallow embeddings for text and visuals, focusing on deep modality interaction through a series of Transformer layers.
• ALBEF (Li et al., 2021) aligns visual and textual features using imagetext contrastive loss first, fusing them via a multimodal Transformer encoder, and employs momentum distillation for enhanced learning from noisy data.
• METER (Dou et al., 2022) employs co-attention for semantic interplay between modalities, featuring layers with self-attention, cross-attention modules, and a feedforward network.
3) The MEL methods include:
• DZMNED (Moon et al., 2018) is a trailblazing MEL method, blending visual, word, and character features through an attention mechanism.
• JMEL (Adjali et al., 2020) utilizes unigram and bigram embeddings for text analysis, merging these features via concatenation and a dense layer. For fair comparison, its textual encoder is substituted with a pre-trained BERT model.
• VELML (Zheng et al., 2022) employs the VGG16 network for object-level visual features and replaces its GRU textual encoder with pre-trained BERT, fusing both modalities using an enhanced attention mechanism.
• GHMFC (Wang et al., 2022a) introduces a hierarchical cross-attention approach for detailed correlation analysis between text and visual features, utilizing contrastive learning for optimization.
• MIMIC (Luo et al., 2023) presents a framework for fine-grained inter-modal interactions, optimized using contrastive learning.

heading: C Statistics of Datasets
text: WikiMEL (Wang et al., 2022a) is collected from Wikipedia entities pages and contains more than 22k multimodal sentences. RichpediaMEL (Wang
et al., 2022a) is obtained form a MMKG Richpedia (Wang et al., 2020). The authors of RichpediaMEL first extracted entities form Richpedia and then obtain multimodal information form Wikidata (Wang et al., 2022a). The main entity types of WikiMEL and RichpedaiMEl are person. WikiDiverse (Wang et al., 2022b) is constructed from Wikinews and covers various topics including sports, technology, economy and so on. Table 5 shows the detail of three datasets. Note that we find there also exist some different entities with the same name, and we report the number of them in the Same Name row in Table 5. Figure 7 displays statistics on the number of text tokens and the count of objects in images across three datasets.

heading: D Metrics
text: We first calculate the similarity scores between a mention and all entities of the KG, then the similarity scores are sorted in descending order to calculate H@k and MRR, which are defined as:
H@k = 1
N
N∑
i
I(rank(i) < k), (34)
MRR = 1
N
N∑
i
1
rank(i) , (35)
where N is the number of total samples, rank(i) means the rank of the i-th ground truth entity in the
rank list of KG entities, I(·) stands for indicator function which is 1 if the subsequent condition is satisfied otherwise 0.

heading: E Details of Variant Analysis
text: Table 6 displays all metrics from the variant experiments on the Richpediamel dataset.

heading: F Analysis on Varying Token Counts
text: We also conducted extensive experiments on samples with different token counts in the Richpediamel dataset. The experimental results are presented in Figure 6. We categorized texts based on the number of tokens: texts with more than 20 tokens were classified as long texts, while those with fewer than 20 tokens were classified as short texts. Short texts contain fewer linking signals, making the short text group more challenging than the long text group. From our observations, we can draw the following conclusions: Our model demonstrates stronger robustness in challenging scenarios compared to attention mechanism and baseline models. In long text scenarios, our model slightly outperforms baseline models and attention mechanisms. However, in the more challenging short text scenarios, while our model only experiences a slight decrease in performance, the baseline models and attention mechanisms show a significant decline.

heading: G Dissusion of the Generative Entity Linking
text: Generative Entity Linking (GEL) (De Cao et al., 2020) utilizes generative models (Lewis et al., 2019; Touvron et al., 2023) to directly generate the target entity of the mention. Due to the end-to-end
paradigm and competitive results, it has received extensive attention (Shi et al., 2023; De Cao et al., 2022; Huang et al., 2022; Yuan et al., 2022).
Generative methods use the constrained decoding mechanism to ensure that the entity names generated by the model must exist in the knowledge graph. Specifically, they construct a Trie of alternative entities, where each leaf in the tree is an entity name. However, they can hardly deal with entities with the same name, which we term as lexical ambiguity problem (Fortuny and Payrató, 2024). For example, computer scientist Michael Jordan and basketball player Michael Jordan share the same leaf node of the Trie, and we find that there are 14 pages (of entities) in Wikipedia called the same name Michael Jordan.
To further investigate the lexical ambiguity problem, we count the number of these entities with the same name in the KG used for the three MEL datasets in the Table 5. We find that the Wikidi-
verse dataset has more data with lexical ambiguity problem (including 14,827 entities), and the Wikimel dataset has fewer data of that (including 616 entities). We further compare the results to the typical multimodal generative GEMEL (Shi et al., 2023), and show the results in Table 7. The results demonstrate that, on the low lexical ambiguity dataset Wikimel, GEMEL is lower than ours, and better than ours on the high lexical ambiguity dataset Wikidiverse. Although GEMEL performs better on the high lexical ambiguity dataset, it neglects the lexical ambiguity problem, thus cannot distinguish the entities with the same name.
Besides, GENER (De Cao et al., 2020), the unimodal generative method of the EL, can also hardly deal with this problem. It tries to solve the lexical ambiguity problem by generating distinguishable entity ids, but the final result drops by about 20% (De Cao et al., 2020). Considering the entity ids contain little semantics, how to generate correct ids is challenging for generative models.
With the emergence of LLMs, the GEL is a promising paradigms nowadays. However, how to solve the lexical ambiguity problem of GEL is one of the most important problems in its application, and we will explore it in the future work.

heading: Future Work
text: However, how to solve the lexical ambiguity problem of GEL is one of the most important problems in its application, and we will explore it in the future work