OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport
Authors: Zongsheng Cao
Qianqian Xu
Zhiyong Yang
Yuan He
Xiaochun Cao
Qingming Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-modal knowledge graph
Representation learning
Optimal transport

Abstract: Multi-modal knowledge graph embeddings (KGE) have caught more and more attention in learning representations of entities and relations for link prediction tasks. Different from previous uni-modal KGE approaches, multi-modal KGE can leverage expressive knowledge from a wealth of modalities (image, text, etc.), leading to more comprehensive representations of real-world entities. However, the critical challenge along this course lies in that the multi-modal embedding spaces are usually heterogeneous. In this sense, direct fusion will destroy the inherent spatial structure of different modal embeddings. To overcome this challenge, we revisit multi-modal KGE from a distributional alignment perspective and propose optimal transport knowledge graph embeddings (OTKGE). Specifically, we model the multi-modal fusion procedure as a transport plan moving different modal embeddings to a unified space by minimizing the Wasserstein distance between multi-modal distributions. Theoretically, we show that by minimizing the Wasserstein distance between the individual modalities and the unified embedding space, the final results are guaranteed to maintain consistency and comprehensiveness. Moreover, experimental results on well-established multi-modal knowledge graph completion benchmarks show that our OTKGE achieves state-of-the-art performance.

OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport
Zongsheng Cao
Qianqian Xu
Zhiyong Yang
Yuan He
Xiaochun Ca
Qingming Huang 4,3,7,8
SKLOIS, Institute of Information Engineering, CAS
School of Cyber Security, University of Chinese Academy of Sciences Key Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS School of Computer Science and Tech., University of Chinese Academy of Sciences Alibaba Group
6 School of Cyber Science and Tech., Shenzhen Campus, Sun Yat-sen University
7 BDKM, University of Chinese Academy of Sciences
Peng Cheng Laboratory caozongsheng@iie ac. cn yangzhiyong21 2uccas2uuucccccc ac cn
xuqianqian@ict cn
heyuan heyuan.hydal hww..hyylllbbba-aa.. baba-inn..... com
caoxiaochun@mail, sysu, edu.
qmhuangducas
Abstract
Multi-modal knowledge graph embeddings (KGE) have caught more and more attention in learning representations of entities and relations for link prediction tasks. Different from previous uni-modal KGE approaches, multi-modal KGE can leverage expressive knowledge from wealth of modalities (image, text, etc.), leading to more comprehensive representations of real-world entities. However, the critical challenge along this course lies in that the multi-modaa embedding spaces are usually heterogeneous. inherent spatial structure of different modal embeddings. To overcome this chal- lenge, we revisit multi-modal KGE from a distributional alignment perspective and propose optimal transport knowledge graph embeddings (OTKGE). Specifically, we model the multi-modal fusion procedure as transport plan moving different modal embeddings a unified space by minimizing the Wasserstein distance between multi-modal distributions. Theoretically, we show that by minimizing the Wasserstein distance between the individual modalities and the unified embedding space, the final results are guaranteed to maintain consistency and comprehensive- ness. Moreover, experimental results on well- establstall multi-men knowledge graph completion benchmarks show that our OTKGE achieves state-of-the-a performance.
this sense, direct fusion will destroy the
Introduction
In the past decades, with the emergence and development of numerous knowledge graphs (KGs) [1, [], spectrum of related applications has been widely facilitated, e.g., question answering [3, 4], semantic search [5, 6), and recommendation systems [7, 8]. Due to the incompleteness of real-world knowledge graphs, link prediction becomes an important procedure for constructing knowledge graphs. To achieve this goal, Knowledge Graph Embedding (KGE) [9, 10, 11, 12, 13) attracts more and more attention. It can learn the low-dimensional representations of entities and relations and thus predict missing links.
Nowadays, most of the KGE methods [11, 12, 13] mainly focus the uni-modal knowledge graphs represented with pure symbols or concepts. However, most data sources we can access exhibit a
"Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

multi-modal property, where information from text, image, and video coexists and comprehensively describes the underlying target. In recent years, researchers [14, 15, 16. 17] point out that knowledge extracted from such multi-modal data can significantly improve the quality of the learned knowledge graph representations. To that end, the study of the multi-modal KGE raises a new wave, and great many meaningful work (18, 16, 19. 15, 20] emerges and achieves remarkable success. Specifically, they usually embed the multi-mode spaces, then fuse these multi-modal embeddings with the help of operations such as concat or mean. The resulting embedding is ultimately employed as a unified representation for the multi-modal entity, However, since the embeddings of different modalities are usually in various heterogeneous spaces, direct fusion might destroy the intrinsic distribution and thus lead to inconsistent and incomprehensive representation in the unified space. Based on this fact, this paper focuses the following question: Can we find new KGE model, which is capable of learning unified representations of multi-modal entities while overcoming the problem of spatial heterogeneity?
knowledge (e.g. texts, pictures) of the entity into different
To address this issue, we propose a novel multi-modal KGE method called Optimal Transport Knowledge Graph Embeddings (OTKGE). Specifically, formulate the fusion procedure of multi-modal knowledge as an opti- mal transport (OT) problem [21]. As shown in Figure 1. OT can move different modal embeddings to a unified aligned space by an optimal transport plan while overcom- ing spatial heterogeneity by minimizing the Wasserstein distance between different distributions. In this way, we can facilitate the information interaction between multi- modal embeddings and obtain unified representation by the Wasserstein barycenter. On top of this, we theoretically demonstrate that the divergence between the distribution of individual source modality and that of the unified space can be bounded by the Wasserstein distance, suggesting that our proposed method can overcome the spatial hetero- geneity issue. Moreover, experimental results demonstrate the superiority of OTKGE on multi-mont modal knowledge graph completion tasks.
oT
Figure 1: mal transport (OT) for multi-modal en- tities. Green and red colors represent the different modalities. Lines represent distributions of different modal entities while circles and triangles represent the embeddings of different modal entities.
An illustration of the opti-
In nutshell, our contributions are summarized three-fold:
For the modal fusion problem in multi-modal KGE, we propose a new fusion approach based on optimal transport, which can effectively tackle the modal spatial heterogeneity by reducing the Wasserstein distance between different modal distributions.
Theoretically, we provide an upper bound for the target errors from source multi-modal spaces to the unified space in OT fusion. It can provide a theoretical guarantee for modal fusion while overcoming spatial heterogeneity.
Extensive experiments show that OTKGE achieves state-of-the-art performance on two multi-modal knowledge graph completion tasks.
2
Related Work
The study of knowledge graphs has attracted widespread attention in recent years. According to the number of used modalities, the previous methods mainly fall into two branches including uni-modal KGEs and multi-modal KGEs.
2.1 Uni-modal Knowledge Greaph Embeddings
In the past decades, researchers have proposed various embedding approaches to achieve uni-modal KGE, named translational distance models, semantic matching models, and neural network models. First of all, translational distance models are derived from the popular method TransE [9], which proposes to model the relation as the translation operation between entities, and adopts the principle head entity++ + relation = tail entity. Subsequently, there are many variants of TransE, e.g., TransH [10], TransR [22] and TransD [23], which model the relation as the projection operation and model

---

the entity as the vectors, respectively. Despite the practical advantages, they cannot model some relation patterns (e.g. symmetric, inverse, and composition) [12].
Semantic matching approaches aim to capture and match latent semantics between entities, which models the plausibility of the triple by semantic matching. For instance, some representative methods are RESCAL [24], DistMult [25]. ComplEx [11], and HolE [26]. It is worth mentioning that RotatE [12] models the relation as the rotation in two-dimensional space, thus realizing the goal to model symmetrr/antisymmetry inversion, and composition patterns. Inspired by RotatE, there are also some variants of RotatE, such as QuatE [13], DualE [27], and ROTH [28], which introduce rotations
into other geometric spaces.
In addition to the conventional methods, there are also some models using neural networks to produce KGE with remarkable effects, such as R-GCN [29]. ConvE [30]. ConvKB [31], KBGAT [32], and A2N [33]. However, some of them are usually computationally expensive and may require pre-train emheddings.
2.2 Multi-modal Knowledge Graph Embeddings
Beyond uni-modal KGE methods, recently a great many multi-modes KGE methods have been proposed, which can be divided into neural network methods and translation methods.
Neural network methods utilize neural networks to learn embeddings for structural and knowledge. For instance, MMRFAN [17] and MKRL [14] adopt the unified space to learn the textual knowledge and visual knowledge of entities and thus realize the multi-modal KGE. Unfortunately, these models such as MMRFAN are designed for some specific cases (e.g., medical knowledge graphs), and they may be not suitable for other general graph structures [20].
Translation methods follow the principle of TransE and model the relation as the translation for multi- modal entities. For instance, DKRL [18] learns the representations of KGs by cxploiting the textual and structural information while IKRL [16] learns representations by integrating visual information and structured information. However, both two models can only one of the textual information and visual information. To tackle this problem. [19) proposes to take the sum of sub-energy functions as the overall score function, which aims to utilize the textual information, visual information, and structured KGs simultaneously. Then TransAE [15] adds a multi-mode autoencoder based on TransE, which can achieve the goal obtain the embeddings of multi-modal KGE. Afterward, MMKRL [20] integrates structured knowledge and multi-modal knowledge in a unified space with the assistance of concat fusion operation. In a word, these models fuse heterogeneous multi-modal spaces by concat or mean operations. neglecting the differences between distributions. Therefore, learning embeddings for multi-modal knowledge graphs is still an open problem to be tackled.
3 Background on Optimal Transport
Before giving an overview of the optimal transport (OT) problem, we first introduce its definition and present a short introduction of the related background. Specifically, OT serves as a way to compare two probability distributions, which can provide a transport plan to transfer one point to another.
Optimal Transport Problem [21]. Suppose we have two set of points X (x),,,
(n) R
y (m)) R with the Dirac (unit mass) distribution denoted as 5(-). Our first goal is to obtain the empirical probability measures je and v/ for X and Y. respectively. To achieve this, we let μ = 1 (x)(((())) and v = Σm ()) where the weight a = (a1, ... ,a,) and B= (B1,-
and Y (1)))
Pm) can be regarded as the probability simplexes. Subsequently, we design ground cost to measure the distance from point (+)) (6) which is denoted as Cij- Further, we need to calculate the transport coupling T between μe and and the T.j can measure the joint probability observing (+) and ( Conveniently, it can be obtained in the procedure of solving the following:
OT(,, Ug C) mint (T, C), where {T IR (xxxm)) [T1m. a, - 1, 33. It is worth noting that (T, C) == tr C' is formulated as the Frobenius inner product of matrices.
Wasserstein Distance. If we formulate the cost function with the p-norm, ie., Cij = (1),(0), 1 then the p-th Wasserstein distance between two empirical probabilistic measure can be defined as
3

---

ext
Text
B1
T₁
Ey
Figure
By
Tv
Unified Representation
T'
Es
Structure
Es
Optimal Transport
Aligned Embeddings
Figure 2: The overall framework for OT fusion. Es, E1, Ev. EF are structural, linguistic, visual, and unified embeddings, respectively. Tı, Tg, Tv are optimal transport plans. Es, Ei, Ey are the according aligned embeddings, respectively. In our method, Ts is an identity matrix and Es Es since the multi-modal knowledge is align with the structural knowledge.
[1/)2// [34, 35, 36). In this paper, we will set = 1, for the sake of convenience, respectively.
Wy(u, u) == OT v;|l
Methodology
In this section, we introduce new KGE method, termed Optimal Transport Knowledge Graph Embeddings (OTKGE) for multi-modal knowledge graphs. First, we present the definition of the multi-modal KGE problem.
Problem Setup. Given multi-modal knowledge graph G (E, R, where & is the set of entities, R is the set of relations, and T= {(h, r, t)lh, E &,r r R} is the set of KG triples. Here the entity is equipped with structural embedding es. linguistic embedding e, and visual embedding ey. Specifically, the uni-modal KGE method only focuses on es while multi-modal KGE can leverage multi-modal embeddings.
Representations for Entities and Relations. An important procedure in the KGE is to perform semantic matching between entities. To this end, previous work mainly focuses on studying how to model the relation and embedding space for uni-modal KGE. In this paper, we take step beyond to leverage the multi-modal knowledge for the representation of entities and relations.
Supposing we have obtained the structural embeddings (by uni-modal KGE or initialization) and multi-modal embeddings (by pretrained models) of the entity in KGs, then here our goal is to learn the unified embeddings. Due the heterogeneity of multi-moda modal spaces, cxists a gap (caused by factors such as distribution discrepancy and dimensional diffcrence) between different multi-modal embeddings. To address such issue, shown in Figure 2, we propose realize the multi-modal alignment by optimal transport, and then perform multi-modal fusion to obtain unified representation.
notice that there
Multi-modal Alignment. Considering that multi-mode molti-modal knowledge is the auxiliary information, let multi-modal embeddings align with the structural embeddings. Here we take the modal alignment between linguistic space I and structural embeddings S as example, and the process between visual space V and structural space S can be achieved similarly.
With the OT alignment strategy, we can model the process of modal alignment a process of transferring different each dimension of modal embeddings into an alignment space (structural embedding space). Specifically, it can be divided into the following steps: 1) Estimate the distribution I for linguistic embedding E, and the distribution for structural embedding Es; 2) Find a transport coupling T from μ v, where Tij gives the probability to transport from the i-th feature dimension

---

of E₁ to the j-th dimension of Es; 3) Use T to calculate the barycenter, and then map Ej to the unified space S and form new embedding Ej.
Following the aforementioned steps, we denote by = {u:J== E A. and - { E Am for the probabilistic simplexes of linguistic and structural embeddings, respectively, Let μ = Σド=1 Uj0 B represent discrete distributions of linguistic and structural embed- respectilly where is the Dirac function. For the sake of convenience, we set u; 1/m, = 1/n, where m, n are the dimensionality of the embeddings in E and Es, respectively. Then we denote the joint distribution [I v): I[(,, v) {T R T1m u, T 1, v). where 1 denotes an all-one vector. Afterward, we compute the OT coupling T between the measures μ and which can be formulated follows:
n
12
WF(n,
min
ΣΣTOC. C.
(1)
TElL(AM)
:=jj=1
where Cij is the cost function that evaluates the distance between Ej and E'y. Moreover, W(p, can measure the distribution discrepancy between E, and Es-
Multi-modal Fusion. After obtaining the transport matrix T, the linguistic embedding E, can be transformed into target-aligned embedding Ê, with the barycenter-baset strategies [34, 35, 36]:
= (1/)(( + E;
(2)
Once this has been done, the transported linguistic embedding E, is the same space as structural embedding Es, where Ar is an adjustable transport parameter [37]. Subsequently, we can obtain the aligned visual embedding Ey in similar way. Then the next step is to fuse the aligned multi-modal embeddings (É, and Ey) with the structural embedding Es utilizing the strategy inspired by [38]:
Ep
min E
1 3
Σ AWW(E, E) (E.V.)
where Ep the unified representation; W(E, E) is the WP distance between the distribution of Eg and that of E; X; represents the weight. The whole procedure is summarized in Algorithm 1.
Algorithm 1: Multi-modal representations fusion.
Input: Distributions (А supported by multi modal representations Eg, and structural distribution / supported by the structural embeddings Es.
Output: Unified representation Ep
Initialize The size of input representation nie the size of fused representation m, the adjustable transport parameter År-
mm/m
foreach E {1, V,S) do N4 1 nu TT
Cs j = Ed Estile Vi E [nn]j E [m]
# [m] means 1,.., m
T (T Cs)
E. diag(L/v)(T; Ex
EF ming? Eie(I.VS)) AWWEE E)
Relational Transformations. On top of the aforemeeniioeed process, we have obtained unified representations of entities. Subsequently, we transform the head entity h in the triple (h, f) with the assistance of
hratation X0 h.
where h is the embedding of the head entity h, is the embedding of the relation hyctation is the embedding of the transformed head entity. Here 6 is the transformations imposed by r, which can refer to some uni-modal KGE mothods, i.e., TransE and RotatE.
5

---

Score Function and Loss. lates as follows:
Having obtained the transformed head entity, the score function formu-
f(h, r, t) = d d(hrrelation t)
(5)
where is the embedding of the tail entity t, and d(..)) represents distance metric (e.g., inner product or Euclidean distance). Afterward, to optimize the parameters, we train the model by minimizing the following loss:
== Σ
log(1 Sep(--label (h,, (h,r,t)))
(6)
(Nu..))DUU
where Yiate! E (-1,1) 1} denotes the label of the triple (h, r. (). Here suppose Ω denotes the set of observed triples, then let Ω E x x & Ω be the set of unobserved triples. the procedure of training, we adopt the negative sampling strategies (e.g. uniform sampling or Bernoulli sampling [10]). Noticing that the multi-modal information already pre-trained in advance, here we update the structural embedding in the training. This is also consistent with our original intention of using multi-modal information to assist KGE.
5 Theoretical Analysis
In this section, we will show that the OT-fusion strategy preserves consistency and comprehensiveness from a theoretical perspective.
First, we introduce the target error [39, 40] as an indicator to measure the distribution relationship between the target unified space and the source space. Here we first give some necessary definitions. Suppose X, 2 and Y' are sets of entities, learned representations, and labels of triples, respectively. Notice that both structural embeddings and multi-modal embeddings of the entity are always assumed to obey the same true score function f 2 - y in multi-modal KGE problems [18, 16, 19, 15, 20]. Considering the true scoring function f is unknown, one usually chooses a predictor function from hypothesis class F for substitution, Vʃ E F, 2 + y. To simplify the notations, here we take the head entity embedding = 2 as the example to analyze while fixing the relation and the tail entity. The conclusion for the tail entity be analyzed similarly.
Afterward, noticing that there exists approximation error between the hypothesis f and the true score function f* under the distribution μ. we measure it here with f)) which is termed target error. For simplicity of notation, we use the shorthand (f)( = f*) Moreover, we utilize the I-Wasserstein distance Wi (-.-) -) to relate the source multi-modal distribution and target distribution. Above all, the target error in the OT fusion can be derived from the following theorem:
(z)])
Theorem (Consistency and Comprehensiveness of the OT Fusion) ) We denote by uss, MI, JV and pF for the distributions of structural, linguistic, visual, and fused unified embeddings. Assume the hypotheses f F are all K K-Lipschitz continuous for some K. Then the following statements hold for every hypothesis f.f E F:F
max iE(L.V.))
{4(f)s S (f()) + 2K
sE(L.V.) max [Wi(p.,FF)
(Consistency)
cr(J) S min (E(V.V
(e(() + (i
(Comprehensiveness)
where Wi(-.)) is the 1- 1-Wasserstein distance, e. (f) is the target error at the corresponding embedding space *.
The proof can refer to Appendix B.
Remark (For Consistency) In the RHS of the inequality, cr(f) is the target error of the unified embedding, which can be optimized implicitly by our training process; max,g(1.y.s) (Wi (pr, pF)} is the maximum of the 1- Wassserstein distance between the individual modality and the unified space, which can be reduced by minimizing the alignment error in Eq.3. Meanwhile, the LHS represents the worst uni-modal performance. In this sense, the theorem above suggests that improving the performance on the unified space can also improve the performance on individual modals. Hence, the learned embeddings are ensured to be consistent with modal-speciff information.

---

Remark 2 (For Comprehensiveness) Similar to the analysis of consistency, the second inequality in the theorem above shows that by minimizing the aligument enror as in Eq.3, is possible to obtain reasonably unifted embedding such that the post-fusion error can be smaller than any involved uni-modal error. This suggests that the OT-fusion strategy can integrate complementary information across different modalities efficiently.
Experiments
6.1
Experimental Setup and Results
Table 1: Statistics of the datasets used in this paper. (Num, represents the number of entities and Num, represents the number of relations.)
Dataset
Num
Num,
Training
Validation
Test
Modality
Scale
WN9-IMG
7k
12k
1k
1k
Multi-modal Multi-modal
Small
FB-IMG
11k
1231
285k
29k
34k
Large
Dataset. terms of the link prediction task, we conduct the experiments and evaluate OTKGE with two standard competition benchmarks as shown in Table 1. There includes WN9-IMG [41] ] and FB-IMG [19]. M9-IMG dataset derived from the subset of WN18 1 [9], which embraces structural knowledge triples, and multi-modal knowledge including textual messages and visual images. FB-IMG dataset is derived from the subset of FB 15K [9), which includes structural knowledge consisting of triples extracted from Freehase [42], and multi-moda knowledge embracing textual messages and visual images.
datasets:
Evaluation metrics. To evaluate our model, we take entities in the knowledge replacing masked entities in the triples, then we rank all candidate triples through the score function. The evaluation metries include the hit rate and the mean reciprocal rank. Specifically, we use H@ n to represent the hit radio with cut-off values 1, 3, 10 while use MRR to represent the mean reciprocal rank in the experiments.
Baselines. We compare our method to some representative models, egg., multi-modal KGE methods including IKRL [16]. TBKGE [19], TransAE [15] and MMKRL [20]; uni-modal KGE methods including TransE [9], DistMult [25] ComplEx [11], and RotatE [12].
Implementation Details. In the course of the experiment, and conduct experiments with single GPU. To be fair, notice that the true triples will get top- rank positions when we conduct the evaluation. In this way, adopt the filtered setting [9] to make sure that all true triples will be filtered out. Moreover, we utilize the grid search to find the hyper-parameter and we choose the best models from the validation set by using early stopping. Specifically, the embedding size i is searched in {100, 200, 400, 500} and the learning rate is searched in f0.001, 0.005, 0.01, 0.05, 0.1].
implement OTKGE2 with PyTorch
The structured embeddings are produced from triples in knowledge graphs, without any external multi-modal sources. To be specific, uni-modal KGE methods such as TransE [9] and ComplEx [11] can be used to learn structured embeddings.
The linguistic embeddings of entities are learned by adopting the word2vec [43] technique. For instance, we learn the linguistic embeddings of FB FB- -MG dataset by pre-trained word2vec while we use GloVe [44] for the WN9-IMG dataset.
The visual embeddings entities are learned by pre-trained VGG [45] models. To be specific, visual embeddings are learned by adopting the (GG-m-118CMM) [46] model in FB-IMG datasets. As for the WN9-IMG dataset, we take the VGG19 [45] model to learn visual embeddings.
hunps:/githubboom//oooo//////77

---

Table 2: Link prediction results the multi-modal datasets (WN9-IMG and FB-IMG). If the indicator of a model is v. it means the model is a multi-modal KGE method; otherwise, is a uni-modal KGE method. OTKGE₁ represents the version with I-Wasserstein distance while OTKGE, represents the version with 2- 2-Wasserstein distance. The best results are bold, and the second-best results are underlined.
FB-IMG
WN9-IMG
Model
Is M
MRR
H@1 1
H@3
H@ 10
MRR
H@!
H@3
H@ 10
TransE
X
.712
.618
.781
.859
.865
.765
.816
.871
DistMult
.706
.606
.742
.808
.901
.895
.905
.925
ComplEx
X
.808
.757
.845
.892
.908
.903
.907
.928
RotatE
.794
.744
.827
883
.910
.901
.914
.26
IKRL
.742
.691
.785
844
.898
.944
.908
.922
TransAE
.755
.698
.794
.857
.901
.900
.912
.928
TBKGE
.812
.764
850
.902
.912
.904
.914
.931
MMKRL
.827
.783
857
906
.913
.055
.17
.322
OTKGE,
.843
.799
.833
.914
.920
.909
.955
.940
OTKGE2
.842
.798
.876
.916
.923
.911
.930
.947
Results on multi-modal datasets. To demonstrate the superiority of OTKGE, we compare it with other popular models on WN9-IMG and FB-IMC datasets shown in Table 2. One can that ОТКGE achieves the best performance on both datasets. On one hand, the uni-modal KGE methods such TransE can only learn the structural knowledge while the rich multi-modal information cannot be used effectively. Therefore, TransE does not perform as well as OTKGE, which demonstrates the advantage of multi-modal KGE. On the other hand, the multi-modal KGE methods such as IKRL use concat or mean to fuse information from multiple modalities, which will be constrained by spatial heterogeneity. Therefore, their experimental performance is weaker than OTKGE, which demonstrates the effectiveness of OT fusion.
6.2 Experiment Analysis
(a) Embeddings in TransE.
(b) Embeddings in IKRL.
(c) Embeddings in OTKGE.
Figure 3: The embedding comparison. Figure 3(a) represents structural embeddings learned by TransE, Figure 3(b) represents
unified embeddings learned by IKRL utilizing concat fusion and Figure 3(c) presents the multi-moda. unified embeddings learned by OTKGE.
The embedding comparison for ОТКGE and other methods. To show that OTKGE (here we adopt the version of 2-Waserrstein distance as an example) can learn a better-unified representation. we choose three kinds of semantic (people, place, and event) entities in the FB-IMG dataset and visualize the entity embeddings shown in Figure 3. these sub-figures, circles blue (people), green (place), and red (event) represent three kinds of semantic embeddings, respectively. On one hand, one can see that using only structural knowledge in TransE can not separate these three kinds of embedding well, which demonstrates the necessity of leveraging multi-modal knowledge. On the other hand, there are still many entities that not clearly separated in IKRL due to the heterogeneity of modal space. Different from this, OTKGE can effectively reduce the Wasserstein distance between

---

different distributions, facilitate the in-depth fusion of different modal knowledge, and thereby learn eetter-unified representation.
The ablation study of components in OTKGE. As shown in Table 3, we conduct the experiments study the role that different components play in OTKGE (here we adopt the version of Wasserstein distance as an example). Specifically, we replace the fusion procedure via optimal transport by the mean and concat operations, which we denote as OTKGE w/ mean and OTKGE w/ concat; We only use the visual knowledge, which we denote as OTKGE w/o ling; we only use the linguistic knowledge, which we denote as OTKGE w/o visual. We can see that the performance of these ablation versions is reduced compared to the original version of OTKGE, which shows that the OT fusion plays an important role in OTKGE
Table 3: Link prediction results on WN9-IMG and FB-IMG datasets.
FB-IMG
WN9-IMG
Model
MRR
H@l
H@3
H@ 10
MRR
H@1
H@3
H@ 10
ОТКGЕ w/ mean
.828
.789
.857
.899
.913
.909
.915
.921
OTКGE w/ concat
.838
.796
.863
.910
.96
.906
.918
.932
OTKGE w/o ling
.835
.791
.867
.908
914
.907
.944
.929
OTКGE w/o visual
.836
.794
.868
.912
.96
.906
.917
.938
Conclusion
In this paper, design KGE model named OTKGE for multi-modal knowledge graphs, where we formulate the fusion procedure by optimal transport theory. Previous work usually neglects the heterogeneity of distributions in the multi-mode modal fusion, which will do harm to the interaction of multi-modal knowledge. To tackle this problem, we propose to transfer the multi-moda information unified space by optimal transport, and fuse the multi-modal information with the Wasserstein barycenter. Theoretically, we prove that OTKGE is advantageous with its capability in learning multi- modal representation and the target error of the multi-moda embeddings and the unified embeddings can be bounded. Moreover, empirical experimental evaluations on multi-modal well-established datasets show that OTKGE achieve overall state-of-the-arr performance.
8 Acknowledgements
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, part by National Natural Science Foundation of China: U21B2038, 61931008. 61733007, U1936208, 6212200758 and 61976202, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDB28000000, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX2021298, and in part by China Postdoctoral Science Foundation under Grant 2022M713101.

---

References
[1] Denny Vrandecic and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Com- ACM, 57(10):78 85, 2014.
[2] Kurt Bollacker. Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collaboratively created graph database for structuring human knowledge. In ACM SIGMOD, pages 1247-1250, 2008.
[3] Denis Lukovnikov, Asja Fischer, Jens Lehmann, and Sören Auer. Neural network-basek question answering over knowledge graphs word and character level. In WWW, pages 121 12200, 2017.
[4] Nouha Othman, Rim Faiz, and Kamel Smaili. Learning english and arabic question similarity with siamese neural networks in community question answering services. Data Knowl. Eng., 138: 101962, 2022.
[5] Chenyan Xiong, Russell Power, and Jamie Callan. Explicit semantic ranking for academic search via knowledge graph embedding. In WWW. pages 1271-1279, 2017.
[6] Duankang Fu, Shufan Zhou, Beijun Shen, and Yuting Chen. Enhancing semantic search of crowdsourcing IT services using knowledge graph. In SEKE, pages 307-409, 2019.
[7] Yixin Cao, Xiang Wang, Xiangnan He, Zikun Hu, and Tat-Seng Chua. Unifying knowledge graph learning and recommendation: Towards better understanding of user preferences. In WWW, pages 151-161, 2019.
[8] Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei- Ying Ma. Collaborative knowledge base embedding for recommender systems. In ACM SIGKDD, pages 353-362. ACM, 2016.
[9] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NeurIPS, pages 2787-2795, 2013.
[10] Zhen Wang. Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI, pages 11 1112-1119, 2014.
[11) Theo Trouillon, Johannes Welbl, Sehastian Riedel, Eric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. ICMI., pages 2071-2080, 2016.
[12] Zhiqing Sun, Zhihong Deng, Jianyun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR, pages 2019.
[13] Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. Quaternion knowledge graph embeddings. In NeurlPS, pages 2731-2741, 2019.
[14] Xing Tang, Ling Chen, Jun Cui, and Baogang Wei. Knowledge representation learning with entity descriptions, hierarchical types, and textual relations. Inf. Process. Manag., 56(3):809- 822, 2019.
[15] Zikang Wang, Linjing Li, Qiudan Li, and Daniel Zeng. Multimodal data enhanced representation learning for knowledge graphs. In IJCN, pages 1-8, 2019.
[16] Ruobing Xie. Stefan Heinrich, Zhiyuan Liu, Cornelius Weber, Yuan Yao. Stefan Wermter, and Maosong Sun. Integrating image-based and knowledge-based representation learning. IEEE Trans. Cogn. Dev. Syst., 12(2): 169-1787 2020.
[17] Yingying Zhang, Quan Fang, Shengsheng Qian, and Changsheng Xu. Multi-modal multi- relational feature aggregation network for medical knowledge representation learning. ACM MM, pages 3956-3965, 2020.
[18] Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. Representation learning of knowledge graphs with entity descriptions. In AAAI, pages 2659-2665, 2016.
10

---

[19] Hatem Mousselly Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. A multimodal translation-bad based approach for knowledge graph representation learning. In *SEM@ NAACL- HI.T, pages 234, 2018.
[20] Xinyu Lu, Lifang Wang, Zejun Jiang, Shichang He, and Shizhong Liu. Mmkrl: A robust embed- ding approach for multi-modal knowledge graph representation learning. Applied Intelligence, pages 1-18, 2021.
[21] Cédric Villani. Optimal transport: old and new, volume 338. 2009.
[22] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI, pages 2181-2187, 2015.
[23] Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. Knowledge graph embedding via dynamic mapping matrix. In ACL, pages 687-696, 2015.
[24] Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In ICML, pages 809-816, 2011.
[25] Bishan Yang, Wentau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In ICLR, pages 1-13, 2015.
[26] Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. Holographic embeddings of knowledge graphs. In AAAI, pages 1955-1961, 2016.
[27] Zongsheng Cao, Qianqian Xu, Zhiyong Yang. Xiaochun Cao, and Qingming Huang. Dual quaternion knowledge graph embeddings. In AAAI, pages
2021.
[28] Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low-dimensional hyperbolic knowledge graph embeddings. In ACL, 2020.
[29] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. ESWC, pages 593-607. Springer, 2018.
[30] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In AAAI, pages 181 1-1818, 2017.
[31] Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. novel embedding model for knowledge base completion based on convolutional neural network. In NAACL-HLT, pages 327-333, 2018.
[32] Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. Learning attention- based embeddings for relation prediction in knowledge graphs. In ACL, pages 4710-4723, 2019.
[33] Trapit Bansal, Da-Cheng Juan, Sujith Ravi, and Andrew McCallum. A2n: attending to neighbors for knowledge graph inference. In ACL, pages 4387-4392, 2019.
[34] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In NeurIPS, 2020.
[35] Sira Ferradans, Nicolas Papadakis, Gabriel Peyré and Sean-François Aujol. Regularized discrete optimal transport. SIAM J. Imaging Sci., 7(3):1853 1883-18888,,, 2014.
[36] Nicolas Courty, Rémi Flamary, and Devis Tuia. Domain adaptation with regularized optimal transport. In ECML, pages 274-289, 2014.
[37] Michaël Perrot, Nicolas Courty, Rémi Flamary, and Amaury Habrard. Mapping estimation for discrete optimal transport.
NeurlPS, pages 4197-4205, 2016.
[38] Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space. SIAM J. Math. Anal., 43(2):004 2011.
[39] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In NewrlPS, pages 3730-3739, 2017.
11

---

[40] Ziyu Liu, Wei Shao, Jie Zhang, Min Zhang, and Kun Huang. Transfer learning via optimal transportation for integrative cancer patient stratification. In IJCAI, pages 2760-2766, 2021.
[41] Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Image-embodied knowledge representation learning. In IJCAI, pages 3140-3146, 2017.
[42] Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collaboratively created graph database for structuring human knowledge. In SIGMOD, pages 1247-1250, 2008.
[43] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
[44] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In ACI. pages 1532-1543, 2014.
[45] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[46] Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. BMVC, 2014.
12

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material URL)? [Yes] In the supplemental material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running exper- iments multiple times)? [No] The experimental results are relatively stable and not sensitive to random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] Assets we use in the experiments are open source.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] Assets we use in the experiments are open source.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "sgzI8B8IUl3", "reviewer": "Reviewer_2fCf", "summary": "The paper studies the multi-modal knowledge graph embedding, which leverages textual, visual, and structure knowledge for learning more accurate representations of entities. The authors propose to utilize the optimal transport to transport different embedding distributions into a unified embedding space, then design a fusion strategy to obtain the final entity embeddings. The authors also provide theoretical analysis to support their claims.", "questions": "- I wonder why OTKGE outperforms other baselines in the uni-modal setting. Is that attributed to the design of relation r? However, we can see from Table 3 that OTKGE w/o trans has minimal performance degradation. It would be better if the authors could provide more detailed ablation studies on uni-model datasets.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The work is well motivated. Multi-modal data have heterogeneous spaces. How to fuse their representations is an interesting problem.\n- It is novel to leverage the optimal transport to transfer different embedding distributions to a unified space.\n- Theoretical analysis is provided to support the central claims.\n- The paper is well written and easy to follow.\n\n\nWeaknesses:\n- Experimental results need more discussion and explanations.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nMvuqaY44HU", "reviewer": "Reviewer_5SJG", "summary": "This paper proposes optimal transport (OT) for the multi-modal fusion procedure of multi-modal KGE. It further provides theoretical analysis on target errors in OT fusion and generalization bounds for multi-modal KGE.", "questions": "1. Will the relationship in Eq. 12 always hold? If not, what are the possible cases that violate Eq.12?\n2. In Line 245-246, the authors said that \"The impact of the intrinsic complexity of function classes will be reduced when the sample size m is larger\", I think this result is quite interesting. Could you provide some experimental results regarding this argument? \n", "limitations": "The limitations of this paper are properly discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. For previous multi-modal KGE methods, the embeddings from different modals are in various heterogeneous spaces, which is hard to avoid by using direct fusion. This paper proposes to mitigate the gap between heterogeneous spaces by using optimal transport (OT). The results of the ablation study clearly show the advantage of using OT for fusion over direction fusion counterparts like average or concatenation for fusion. Moreover, OTKGE can capture a wider range of Inference patterns than previous works.\n2. Theoretical analysis of two important questions is given. Theorem 1 shows the target error can be bounded by the Wasserstein distances of different modals. Theorem 2 shows the generalization bound for the latent representation quality, and it further implies that multi-modal KGE can outperform uni-modal KGE analytically. In addition, the guarantee of Theorem 1 can not be easily achieved by other methods due to their fusion approaches. Theorem 2 justifies why multi-modal KGE is better. I think these theoretical results well distinguish the proposed methods from previous methods.\n3. The experimental results are comprehensive, and the ablation study aligns with theoretical analysis.\n\nWeaknesses\n1. The organization of the theoretical analysis can be improved. It's better to split section 4 into two subsections. In addition, it would be better if the authors could state the relationships between Theorem 1 and Theorem 2 (if there are any). In addition, algorithm 1 and algorithm 2 can be moved into the main text instead of the appendix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e1VKGCggCE7", "reviewer": "Reviewer_K5Tr", "summary": "This paper focuses on the multi-modal fusion problem. The authors propose a method to align the different modalities by reducing the Wasserstein distance between different modal distributions. Experimental results show that the proposed method achieves state-of-the-art performance.", "questions": "1 In Equation (4), the symbol E represents the structural embedding? The author should clarify.\n2 In the experiment, the author should compare the proposed fusion method with other multi-modal fusion methods based on the aligned multi-modal embeddings.\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1 The paper is well written and easy to follow. \n2 The motivation is clear.\n3 The experiments and ablation studies well demonstrate the effectiveness of the proposed method.\nWeaknesses：\n1 The idea of aligning the different modalities is not novel.\n2 Equation 4 is supposed to be further explained. What’s the insight behind the multi-modal fusion based on the Wasserstein barycenter?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept