Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
Authors: Yichi Zhang
Zhuo Chen
Lingbing Guo
yajing Xu
Binbin Hu
Ziqi Liu
Wen Zhang
Huajun Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-modal Information Fusion
Knowledge Graph
Multi-modal Entity Representation
Mixture-of-Experts
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Learning semantic-rich multi-modal entity representations in the knowledge graphs with mixture of multi-modal knowledge experts.

Abstract: Learning high-quality multi-modal entity representations is an important goal of multi-modal knowledge graph (MMKG) representation learning, which can en- hance reasoning tasks within the MMKGs, such as MMKG completion (MMKGC). The main challenge is to collaboratively model the structural information concealed in massive triples and the multi-modal features of the entities. Existing methods focus on crafting elegant entity-wise multi-modal fusion strategies, yet they over- look the utilization of multi-perspective features concealed within the modalities under diverse relational contexts. To address this issue, we introduce a novel framework with Mixture of Modality Knowledge experts (MOMOK for short) to learn adaptive multi-modal entity representations for better MMKGC. We design relation-guided modality knowledge experts to acquire relation-aware modality embeddings and integrate the predictions from multi-modalities to achieve joint decisions. Additionally, we disentangle the experts by minimizing their mutual information. Experiments on four public MMKG benchmarks demonstrate the outstanding performance of MOMOK under complex scenarios. Our code and data are available at https://github.com/zjukg/MoMoK.

Published as a conference paper at ICLR 2025
MULTIPLE HEADS ARE BETTER THAN ONE: MIXTURE OF MODALITY KNOWLEDGE EXPERTS FOR ENTITY REPRESENTATION LEARNING
Yichi Zhang Zhuo Chen 1.2, Lingbing Guot:2, Yajing Xu Binbin Hu', Ziqi Liu
Wen Zhang 4.2; Huajun Chen 1.2.5+
'College of Computer Science and Technology, Zhejiang University
PZJU-Ant Group Joint Lab of Knowledge Graph AAn Group
"School of Software Technology, Zhejiang University
*Zhejiang Key Laboratory of Big Data Intelligent Computing zhang. wen,
Izhangyi chi2022,
ABSTRACT
Learning high-quality multi-modal entity representations is an important goal of multi-modal knowledge graph (MMKG) representation learning, which can en- hance reasoning tasks within the MMKGs, such as MMKG completion (MMKGC). The main challenge is to collaboratively model the structural information concealed in massive triples and the multi-modal features of the entities. Existing methods focus crafting elegant entity-wise multi-modal fusion strategies, yet they over- look the utilization of multi-perspeen under diverse relational contexts. To address this issue, introduce novel framework with Mixture of Modality Knowledge experts (MoMoK for short) to learn adaptive multi-modaa entity representations for better MMKGC. We design relation relation-guided guided modality knowledge experts to acquire relation-aware modality embeddings and integrate the predictions from multi-modalities to achieve joint decisions. Additionally, we disentangle the experts by minimizing their mutual information. Experiments on four public MMKG benchmarks demonstrate the outstanding performance of MoMoK under complex scenarios. Our code and data are available at / com z jukg/MoKKK
features concealed within the modalities
INTRODUCTION
Multi-modal knowledge graphs (MMKGs) (Chen et al. (024) are an extension of traditional knowl- edge graphs (KGs) (Liang et al.) 2024b), encompassing rich modality information such as images and textual descriptions of scage entities, which bridges structured knowledge triple and unstructured multi-modal content together. Based on such a specialized and expressive data organization format, MMKGs have evolved the emerging infrastructure of Artificial Intelligence (AI), contributing to numerous Al-related fields like large language models (Zhang et al.
2023b). recommendation systems (Wang et al. 2019a and other practical applications (Chen et al.
2023a Liang et al. 2024a).
Learning better entity representation is a crucial topic in the representation learning and KG commu- nity. It aims to encode the complex information in the given KG to perform KG reasoning tasks like knowledge graph completion (KGC) (Liang et al. (023). which is an interesting and important task seeking to automatically discover new knowledge from the existing KGs. KGC allows for the missing entity prediction to given entity-relation query, e.g., (ICLR 2025, Located In, ?). Traditional KGC usually learn entity representations by modeling the triple structure in the embedding space. As for MMKG, the situation becomes more complex. Multi-modal knowledge graph completion (MMKGC) further enhances the entity embeddings with multi-modal features, aiming to collaboratively model the triple structure and mudti- content to achieve robust prediction.
Existing MMKGC methods (Wang et al. 2021 Xu al., 2023) typically employ multi modal fusion module to integrate the information from different modalities to obtain joint entity em-
"Corresponding Authors.

---

Published as a conference paper at ICLR 2025
beddings. These entity embeddings are then mapped into scalar score along with the relation embeddings as a basis for assessing the triple plausibility. MMKGC, being a prediction task in multi-relational scenario, is influenced by different relational contexts. which in turn affect the selection and utilization of entity modality features. As illustrated in Figure [ different sections of varied modality information emphasize their respective significance when making predictions based on different relationships. However, such a conventional paradigm overlooks the informa- tion diversity both inter-modality and intra-modality. Different modalities can represent various aspects of entity information, and information within the same modality can also play different roles depending on the relational context. If vanilla multi-modal fusion is performed directly at the entity level without considering the relational context, it can result in low utilization of this mudti-modal information and finally learn immutable entity embeddings across different re- lational contexts, thereby limiting the model's performance. This limitation is particularly pro- nounced in realistic scenarios where many entities are subject to modal noise and incomplete information, which can further challenge the model's ability to utilize modal information effeettvvlly
To address these issues, propose Mixture of Modality Knowledge experts (MoMoK) frame- work in this paper. MoMoK incorporates relation-guided modality knowledge experts for each modality, which constructs expert networks in each modality. These expert networks, guided by the relational context of the current triple, adaptively aggregate the multi-view embeddings for entities. Further, MoMoK employs multi- modal joint decision to integrate the modality embeddings as a new joint modality and achieve comprehensive triple prediction in an ensemble manner. Ultimately, we employ expert information disentanglement module to differentiate learning across different expert networks with contrastive mutual information estimation, aiming to force different experts specialize in different relational contexts. This entire process be likened to each modality functioning as a senior expert, gathering the insights of junior experts within the corresponding modality, based on the principle that multiple heads are better than one. These insights are then communicated and integrated across modalities to facilitate more comprehensive decision-making. We conduct comprehensive experiments on four public MMKG benchmarks to demonstrate the effectiveness of our framework MoMoK with further exploration to validate its properties. Our contribution can be summarized as:
Modality Information len Ichthyasuuri
Text Modaliy
Image Madality
Lchthysssers
orggn
somatimes referned Nchhttyoouret althugh the tern olee
hos nos_ollos
Rood for wider cludes in which the ender
rnsioss Ichthyoosuiios rrind Etrvved dunig
Figure I: Different relational context requires dif- ferent modality information for proper prediction.
We address the problems in modality information utilization by MMKGC models and propose MoMoK with relational-guice modality experts and multi-modal joint decision to learn better entity representations and unleash the power of multi-modal information in MMKGs
We examine the learning of different modal experts through the lens of mutual information estima- tion, and propose to decouple and discretize the expert information within a modality using mutual information comparison estimation. We present detailed theoretical analysis to justify the design.
We conduct extensive experiments against 20 recent basclines on four MMKG benchmarks to demonstrate the state-of-the-e (SOTA) performance of MoMoK and further explore its robustness, reasonability, and interpretability in the complex environments.
RELATED WORKS
Multi-modal Knowledge Graph Completion (MMKGC) MMKGC (Chen et al. 2024) aims to automatically discover new knowledge triples from the existing MMKGs by collaboratively modeling the triple structure and multi-modal information (c.g.. images and textual descriptions) in the MMKGs. Mainstream MMKGC methods (Xie et al. 2017) explore multi-modal fusion in the same representation space to measure the triple plausibility from multi-views. Advanced multi- modal fusion techniques such as optimal transport (Cao et al.) (022), modality ensemble (Li et al. (023), contrastive learning (Liang et al. 2024c), and adversarial training (Zhang et al. (024a) have been continuously introduced into MMKGC. Due to the page limit, a detailed introduction the existing MMKGC methods can be found in Appendix
2

---

Published as a conference paper at ICLR 2025
Our MoMoK Framework
(I). Relation-guided Modality Knowledge Experts (ReMoKEs)
(IT). Muti-moda Joint Decision (MuJoD)
Training
Shrushurs
Vinual Locs
Textunl Loss
Johnt
Objactive
Locs
Loss
Holta
Expert
Modailty
Some
Sooee
Expert2
1
Jesul Srieoding
Expert X
Fiilon
Octpus
Modaliyy
Outpur
Adaptive Fusion
Kezwdedge Eeperts
Pecoon
f
Rabtokt Outputt
(IIT). Expert Information Disentonglement
Structure
Visual
hoo
Expert
1
Verittondl
CLUB
Meddlity
Network
Paslerio kor
Divergente
Losa
Hull-moddsi
Structure
Image
Figure 2: Overview of our proposed MoMoK framework, which consists of three core components: the caation-guided modality knowledge experts (ReMoKE), multi-modal joint decision (MuJoD), and expert information disentanglement (ExID).
Mixture-of- -fpprtssss-tssssssseooo (MoE) MoE is special model ensemble and combination method that is widely used in AL-related fields like computer vision (Chen al., 2023b) Riquelme et al.) (002I) natural language processing (Jiang al. 2023 Zhao et al. 2024 (025), recommendation (Ma et al.) 2018: Bian al., 2023) Hou et al. 22022), , and so on. MoE usually divides given task into multiple subtasks to solve them with individual expert models and design a routing module to select suitable experts to solve the current task. The MoE architecture creates a buzz due to its successful use in large language models (LLMs) (Jiang et al. 2023) as it can efficiently train larger and stronger LLMs.
3 PROBLEM DEFINITION
general KG can be formalized as KG= (E, R. T) where &, R. are the entity set, the relation set respectively. T r,, h,t E,r E R. is the triple set. Furthermore, MMKGs have modality set denoted as M, encapsulating different modalities in the MMKGs. For an entity E E, its modality information of modality m E .M is denoted as (e). For different modalities, the elements in it have different forms. For instance, Xm (e) can be a set of images for image modality and some video clips for video modality. Note that the triple structure (S) is also an extra modality where the structural information is embodied in the triple set T.
R + to discriminate the plausibility of given triple (h, r. t). In this context, higher score implies more plausible triple. Entities and relations are embedded into continuous vector spaces for data-driven learning. For MMKGs, all the modality information of each entity will be represented modality embeddings e.. (m E .M) to participate the triple score calculation by multi- modltilmddal fusion and integration. During training, negative sampling (NS) is widely used construct manual negative triples for contrastive learning because KGs only have observed positive triples. The negative triples are denoted as:
MMKGC designs score function S(h, r, 1)
T' T'=(("',r; ((h',,,)) (,, TTnl' E E - [hh} U {(h, r, (') (h,r, E Tn' E (4} (1)
which is generated by random replacement of entities in the positive triple. During inference, the MMKGC model is usually evaluated with the link prediction task (Bordes et al. 2013) to predict the missing head or tail entity the given query (?,r, r, I) or (h, r, ?). For each candidate & E, the score of the triple (h, e) (e,r, () is calculated and then ranked across the entire candidate set.
4 METHODOLOGY
In this section, we will present our proposed framework called Mix of Modality Knowledge experts (MoMoK for short) to achieve robust MMKGC. There three key components design:
3

---

Published as a conference paper at ICLR 2025
relation-guides modality knowledge experts (ReMoKE for short), mudti-modal joint decision (MuJoD for short), and expert information disentanglement (ExID for short).
4.1 RELATION-GUIDED MODALITY KNOWLEDGE EXPERTS
To better learn the embedding of different perspectives intra- moda-modal,,ies,, we introduce module called relation-guiie guided modality knowledge experts (ReMoKE) to build expert networks in each modality. First, for each modality m E M, the entity e & possesses raw modality feature 6m. derived from the modality data Xm (e). For image and text modality, a pre-trained model like VGG (Simonyan & Zisserman 2015) and BERT (Devlin et al.) 2019) would be employed extract the raw modality feature. As for the structure modality, the raw modality feature will be learned from scratch with the triple data during training.
We then learn the multi pulti-pespective embeddings V Ve m.2.-... VE m. K of the entity and modal- ity m by establishing K modality knowledge experts (MoKE) for each modality denoted as Wm,1, 2m, Wm.K. This process can be represented as V (em.: Then we de- sign relation-guided gated fusion network (GFN) to facilitate intra-modality entity embedding fusion with relation guidance. The output entity embedding for modality m and relation is denoted
as Cm.r
- G2(Vm, r),, where G2 is the weight for each MoKE calculated by the GFN:
exp (Aum(V (Vi,.) om))))/((EF)
G((mm,::)) Σj=1 exp (14m() + om.))/((Er))
where 6m, (0, (VR.)) (2)
where Un are two projection layers and 6m, is tunable Gaussian (Shazeer et al. 2017 noise to balance the weights for each MoKE and augment the robustness of the MMKGC model. This is design (Shazeer et al.) (017)) that has been proven to work. Besides, we add learnable relation-aware temperature E, with sigmoid function to limit the temperature in the range (0, 1). We aim to acquire an entity modality embedding within the relational context of the current prediction before making the final decision. This approach allows modality embedding learning in MoKEs, thereby enabling our MoKEs to extract relation-aware modality embeddings. Besides, the MoKEs will be differentiated to adapt different relational contexts with the design of GFN. We can learn dynamical modality embeddings of entities that change different relational contexts.
introduce the relational context to guide the
4.2 MULTI-MODAL JOINT DECISION
With the ReMoKE module, we can obtain relation-guided modality embeddings (m.-(m E M) for each entity under relational context. Subsequently, we equip the model with the ability to amalgamate information from various modalities to facilitate joint picision-making via MuJoD module. MuJoD first accomplishes multi-modal entity embedding fusion by learning a group of adaptive weights for each entity as:
exp(Wattn O Pm pm((m.))
é Joint.r ΣREM exp(Wattn ④ Pn(Cnr)) Pm (Em,r)
where Pmm (m E M) is a projection layer for modality transformation, Wattn is a learnable attention vector shared by each modality, and @ is the product operator. The joint embedding Jontt aggregates information from all modalities and we treat it as another new "modality" J (short for joint).
We further employ Tucker (Balazevic et al. 2019) score function Sm M) measure the triple plausibility from each modality's ' perspective, which is denoted as:
Sm, rh, Wm x1 hm,r X2 T' X3 tm.r
(4)
where X; represents the tensor product along the i-th mode, T' is the learnable embedding of relation r for each modality, W., is the core tensor learned during training. We train our model with cross-entropy loss for each triple. For given triple r,,r,,), 4), we treat as the golden label for tail prediction against the whole entity set and h as the golden label for head prediction, which is the M U (J}
negative sampling process mentioned before. The training objective of each modality can be denoted as:
log exp(Sm. r,)) + log exp(Sm (.,mhhrr (h,r, Eive exp(Smm (N,,,,, Evee exp(Sm (m,
Lm = Σ
(5)
(h.T.))T
4

---

Published as a conference paper at ICLR 2025
This is the standard KGC model training objective, which MuJoD extends to train separate scoring function for each modality. The overall training objective of MuJoD can be denoted as:
Lkge
E Lm
(6)
MEMU{J}
Since these objectives from different modalities have consistent prediction targets, we directly combine them to derive the final loss Lkge. In the design of MoMoK, construct intra-modality experts to learn caation-guided embeddings in ReMoKE, and further collectively combine these inter- mnder-modality decisions to make expert, making decisions in collaboration with the insights from the intra-modality junior experts (single networks ReMoKE). This hierarchical expert network architecture enables the progressive delivery of valuable entity modal information.
thoughtful predictions. Each modality serves senior
4.3 EXPERT INFORMATION DISENTANGLEMENT
Additionally, to further allow the model to learn multi-perspective embeddings guided by the relational context, propose another expert information disentanglement (ExID) module to disentangle the experts' decisions in each modality based on contrastive log-ratio upper bound (CLUB) (Cheng et al., 2020), which minimizes the mutual information between the multi-perspective embeddings for each modality using CLUB. The general formulation of CLUB can be denoted as:
ICI.UB(;; y) Ep(a..) [log [)] (pp())pp())) [log p(y x)]]
where æ: y are two random variables. (ClUB( y) is upper bound estimation of their mutual information 1(x; y). By optimizing such an objective, we can achieve information disentangling between æ and y. Since the conditional probability distribution p(y x)) is difficult to observe, we can use a variational approximation network to approximate this probability distribution. For the above analysis, we provide more detailed formula derivations and proofs in the Appendix B In our implementation. for modality m with K MoKEs, we disentangle the multi culti-persppecivee embeddings (1 S < K) of MoKEs from each other by the following CLUB mutual information estimation:
Lolub
1
ΣEEΣ
log Qa (Vi,, (mm))
Σ
log Qa 1
(8)
K2
MEMeEB i=1
e
where B is a batch of entities and (e.m. is the variational approximation of ground-truth posterior of y given x parameterized by neural network 0 for modality m. e is another entity sampled from the batch B. With such contrastive loss, we can then make MoKEs minimize the mutual information between decisions. Meanwhile, Qa.m. should also be trained to o minimize the KL-divergence between the real conditional probabilities distribution P(Ve.) (1 DV and the variational approximation Qq (Ve (mm.l Dee m,.) by optimize the Kullback-Leebberr divergence:
Lexid DEL [P(V (Vm..)) 1 (Ve..)) ( [m..)]
which will be alternatively optimized with the main MMKGC model during training. Here, the real conditional distribution usually assumed as Gaussian distribution (Cheng al. 2020).
4.4 TRAINING AND INFERENCE
Combining all the designs above, the final objective for MMKGC model can be represented as:
L Lxge + XLetub
(10)
We collectively train the embeddings with prediction losses C. from each modality in a multi-task manner. The disentangle loss Lotub is regulated by weight X. Besides, during each round of training. @a. is also optimized with the loss Lezid, scparated from the MMKGC model. During the inference stage, calculate the joint score for each triple as S(h, r,t) EmemuçJy Sm (h, r, t) which considers the contribution from each modality and provides full-view prediction. This score function S(h, r. t) will be the final measurement the triple plausibility and applied for candidate triple ranking and evaluation.
5

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS AND EVALUATION
In this section, we will introduce the basic experiment settings of our work and demonstrate our evaluation results with extensive analysis. The following five research questions (RQ) are the key questions that we explore in the experiments.
RQL. Can MoMoK outperform the existing baselines and achieve state-of-the-art performance?
RQ2. Can MoMoK maintain robust performance tasks when the modality information is noisy?
RQ3. How much do each module the MoMoK contribute to the final performance?
RQ4. How does the training efficiency of our model compare to existing methods?
RQS. Are there intuitive cases to straightly demonstrate the effectiveness of MoMoK?
5.1 DATASETS
We conduct our experiments on four public MMKG benchmarks: MKG-W W (Xu al.
2022), MKG-Y
2024a). MKG-V W and (Xu et al., 2022). DBI5K (Liu et al. 2019). and KVC16K (Zhang et al.
MKG-Y Y are the subsets of Wikidata ( Vrandecic & Krötzsch 2014). YAGO Suchanek et al. 2007). and DBPedia (Lehmann al., 2015) respectively. KVC16K is modified from KuaiPedia (Pan et al., 2022), a micro-video encyclopedia. They are all real-world MMKGs with image and text modalities.
The detailed information on the datasets can be found in Table in Appendix.
5.2 EXPERIMENTAL. SETTINGS
Baseline Methods To make comprehensive comparisons, we chose 20 recent SOTA MMKGC
methods as the baselines for the experiments. The first category is uni-modal KGC methods including TransE (Bordes et al.) (013). DistMult (Yang et all]|221111 2015)) ComplEx (Trouillon et al. (016). RotatE (Sun et al.j 2019). PaiRRE (Chao et al. (021). and TuckER (Balazevic et al.) (019)). The second category is mudti-modal KGC methods considering multi-modal information of entities to enhance the (017), TBKGC (Sergieh et al., 2018), TransAE (Wang et al.
KGC models, including IKRL (Xie et al.
2019b), RSME (Wang et al., (221)). MMKRL (Lu et (002)). VBKGC (Zhang & Zhang 2022). OTKGE (Cao et 2002). MoSE (Zhao et al. 2022). MMRNS (Xu et 20.2). MANS Zhang et al. 2023a), IMF (Li et al. 2023). QEB (Wang et al. 2023). VISTA (Lee et al. 2023), and AdaMF (Zhang et al. 2024b). Among these methods, MoSE and IMF are two methods using ensemble learning technologies, which have similarities to our design and are worth making comparisons. The
third category negative sampling methods including MANS (Zhang et al., 2023a) and MMRNS
(Xu et al. 2022 Please refer to Appendix C. for more detailed information.
Task and Evaluation Protocols We evaluate the MMKGC models with the link prediction task (Bordes et al., (2013). the most popular KGC task. We use rank- based metrics like mean reciprocal rank (MRR) (Sun et al. 2019). and Hit@K K (K=1, 3. 10) et al.) (013) to quantitatively evaluate the link prediction performance, considering both head prediction (h, r, ?) and tail prediction (?r, t). Filter setting (Bordes et al., 2013) is used to eliminate the effect of triples that have already appeared the training data. Please refer to Appendix C.2 for more detailed information.
Implemention Details In our experiments, we implement our method with PyTorch and conduct each experiment on a Linux server with the Ubuntu 20.04.1 operating system and a single NVIDIA A800 GPU. The variational approximation network and the projection layers are all implemented by two-layer MLPs with ReLU as activation (Glorot et al. (011). During training, we set the batch size to 1024. The embedding dimension d is tuned from {200, 250, 300, 400, 500). ). We optimize the model with Adam (Kingma & Ba 2015 and the learning rate is tuned from {1e-3 5e le-1). '). The loss weight is tuned in {Ie Ie 11e5) The number K of experts in ReMoKE is set to 3. Each experiment takes 1-3 hours to accomplish across different datasets. For baselines, we reproduce the results following the settings described in the original papers and their open-source official code.
Some of the baseline results refer MMRNS (Xu al. 2022

---

Published as a conference paper at ICLR 2025
Table 1: The main MMKGC results four datasets. The best results bold and the second best results are underlined. Methods with special mark are ensemble-based methods.
MKG-W
Model
MRR
Hit1
MRR
Hit@
MRR
Hiee
H
Hit@
MRR
Hit@
Hit@3
Hit@ 10
Uni-modal KGC Methods
TrunsE
29.19
21.06
30.73
23.45
24.36
12.78
31,48
47.07
8.54
0.64
10.97
23.42
DistMult
20:99
15.93
25.04
19.33
23.03
14.78
26.28
39.59
6.37
3.03
6.11
12.61
ComplEx
7.48
RotatE:
33.67
26.80
34.95
29.10
29.28
17.87
36.12
49.66
14.33
8:25
15.37
26.17
PairRE
34.40
28.24
32.01
25.53
31.13
21.62
35,91
49,30
15.90
9.79
17.24
Mulli-wodal KGC Methods
IKRL
32.36
26.11
5.42
TBKGC
31.48
25.31
33.99
30:47
28.40
15.61
37,03
49.86
5.39
0.35
5.04
15.52
TrunsAE
30.00
21.23
28.10
25.31
28.
21.25
31.17
41.17
10.81
5.31
11.34
21.89
MMKRI.
30.10
22.16
26.81
3.89
18.34
RSME
29.23
23.36
34.44
31.78
29.76
24.15
32.12
40.29
12.31
7.14
13.21
YBKGC
30.61
24.91
37.04
33.76
30.61
19.75
37.18
49.44
14.66
8.28
15.81
27.04
OTKGR
34.36
28.85
35.51
31,97
23.86
25.89
34.23
8.77
5.01
15.55
33.34
27.78
36.28
33.64
28.38
21.56
30.91
41.67
8.81
4.75
9.46
16.40
IMF
34.50
28.77
35.79
32.95
32.25
24.20
36.00
48.19
12.01
7.42
12.82
21.01
QEB
32.38
25.47
34.37
29.49
28.18
14.82
36.67
51.55
12.06
5.57
13.03
25.01
VISTA
32.91
26.12
30.45
24.87
30.42
22.49
33,56
45.94
6.97
12.66
21.27
AdaMF
34.27
27.21
38.06
33.49
32.51
21.31
39.67
51.68
15.26
8.56
16.71
28.29
Negotive Sampling Methods
MANS
30.88
24.89
29.03
25.25
28.82
16.87
36.58
49.26
10.42
5.21
11.01
20.45
MMRNS
35.03
28.59
35.93
30.53
32.68
23.01
37,86
51,01
13.31
7.51
14.19
24.68
MoMoK
35,89
30.38
37.91
39.57
43,45
54.14
16,87
10.53
18.26
29.20
Improvveennts
+2.5%
+1.2%
+3.9%
+21.1
+33.8%
+9.5%
+1.8%
+10.6%
+23.0%
+9.3%
+3.21%
5.3
MAIN RESULTS (RQ1)
The main MMKGC results are detailed in Table Comparison with the recent baselines reveals that MoMoK makes significant progress in almost all the metrics and achieves new state-of-th offtc--------rrrt-- results. When contrasted with existing ensemble-based approaches such as MoSE (Zhao al. (2022) and IMF (L. et al. 2023), MoMoK excels by fully exploiting the potential of the relational context. These methods often merely assign weights to models across different modalities, overlooking the impact of intricate factors like relational context. In contrast, MoMoK thoroughly incorporates these considerations.
Furthermore, it evident that MoMoK achieves most pronounced improvements in Hit@ across different metrics. For instance, MoMoK obtained 33.8% / 23.0% relative improvement of Hit@ on DBI5K and KVC16K respectively. This underscores that, compared with baseline models. our method is more effective at ranking correct answers first. It demonstrates the significant contribution of our multi-modal information utilization and relational context the refined, accurate reasoning capabilities of the MMKGC model.
5.4 MMKGC EXPERIMENTS IN COMPLEX ENVIRONMENTS (RQ2)
To assess the robustness of our method in complex scenarios, we conducted MMKGC experiments in three different scenarios: modality noisy scenario, modality missing scenario, and link sparse scenario. For the modality noisy scenario, we set a noise ratio for the MMKG dataset, according which portion of the solid modal information is added with Gaussian noise before performing the MMKGC experiments. For the modality missing scenario. we randomly remove some of the modality information for certain proportion of entities. For a link sparse scenario, we randomly remove some of the training triples and create a data-sparse environment.
As presented in Figure 3] the experimental results indicate that MoMoK continues outperform the baseline method despite these conditions. It is evident that the complex environments significantly affect the MMKGC prediction at coarse grains, as indicated by the more pronounced volatility of Hit@ 10 compared to MRR. The variation in Hit@ 10 results reveals that baseline methods like TBKGC (Sergich et al., 2018) and AdaMF (Zhang et al. 2024b) undergo noticeable performance degradation with increasing noise and decreasing training data, while method's performance remains relatively steady. From the above implementation results, our approach achieves better results compared baseline in variety of complex scenarios.
7

---

Published as a conference paper at ICLR 2025
-4-
MB2
NIOR
14
13%
30%
30%
4/s
30%
15% Spars
Mising
>5
10%
20%
70%
X0%
4P%
10%
19%
30%
Nolsy
Misinn
Sparse
MoMoK
AGMMP
OBE
TDKGC
43 Spane Inke Figure 3: MMKGC results (MRR and @it@ 10) of DB15K dataset under three different scenario: modality noisy, modality missing, and link sparse. We compare our method MoMoK with three recent MMKGC baselines AdaMF, TBKGC, and QBE.
(3), Medality Nonay
(3), Madality Mivaing
Table The ablation study results on MKG-W and DB15K datasets. We explored the impact of the design of each modality already each important component the final result.
MKG-W
DB15K
Setting
MRR
Hit@1
MRR
Hit@l 1
Hit@3
Hit@ 10
Full Model
35.89
30.38
39.57
32.38
43.45
54.14
(1.1). Structure Modality
32.82
27.73
36.45
29.36
39.99
49.86
Modality
(1.2). Image Modality
32.75
27.78
36.84
29.80
40.10
50.42
Contribation
(1.3). Text Modality
32.62
27.66
37.04
29.93
40.49
50.39
(1.4). Joint Modality
34.76
29.33
36.87
29.90
42.44
53.93
(2.1). w/o relational e,
35.50
29.98
39.40
31.47
43.19
52.88
(2.2). w/o noise 6.,
35.31
29.69
39.43
31.54
43.32
53.75
Model Design
(2.3). w/o udaptive fusion
35.34
30.04
39.01
30.74
43.29
53.85
(2.4). w/o joint training
32.73
27.09
37.62
29.72
41.64
52.73
(2.5). ExID
29.49
38.42
30.63
42.42
53.24
5.5 ABLATION STUDY (RQ3)
To confirm the soundness of our design, we conduct further ablation studies investigate the contribution of each module in MoMoK. Our ablation experiments are divided into two main parts. The first part aims analyze the information from each modality and validate whether they positively contribute to the performance. The second part is dedicated examining designs in MoMoK (ReMoKE, MuJoD, ExID) and verifying whether their design has rationality by removing the corresponding modules. The experimental results presented in Table
50
Hings
Hing
MRX
Haspt
NENKK Revalls
NOIKKC Rosults
45
40
35
0,06001
6.801
Figure 4: Additional parameter analysis about the number of MoKEs and the weight À for the Letub
8

---

Published as a conference paper at ICLR 2025
Table 4: A case study of MoMoK. We list of the relations that predicted best by each modality score Sm to verify the contribution of each modality to the final result.
Modality
Outperforming Predictions
Structure
RoutcJunction. PartF,. Nearest City, Publisher, PrimeMinister, ComputingPlattorm LargestCity SisterStation, League, Parent, HubAirport, Company, Owner, Capital SisterStation, Puhlisher, Head, FoderalState, Parent, ComputingPlatfon CountrySeat GovemingBody, PurtOf, Creator. Company, CaaputiigPlattorn RegionServed SisterStation, RouteJunction, GoverningBody, Publisher, PartOf, FederalState, Computingllattomm, Parent
Image
Text
Joint
Full Model
From the first group of experimental results observe that each modality's information con- tributes to the final result, and during training we set up separate model for each modality, with datasets being lower than the full model result.
their respective performance
Moreover, the results from the second group reveal that some of our key designs in the three modules significantly contribute the final performance. Experiments (2.1) and (2.2) confirm the effectiveness of relational context and tunable noise in the ReMoKE module. Experiments (2.3) and (2.4) focus on the MuJoD module and the results proved the effectiveness of the adaptive fusion (Equation 3 and joint training (Equation 6). Experiment (2.5) further examines the impact of the CLUB loss on information disentanglement Collectively, these findings indicate that joint training has the most profound effect on the final performance, as it trains a separate MMKGC model for each modality, resulting in decision fusion.
We also investigate the effect of some crucial hyperparameters, such as the number of experts K in the ReMoKE module, and the weights of the ExID loss X, as depicted in Figure can be observed that the impact of the number of experts K on the final results generally follows pattern of initial increase followed by a decrease, mainly affecting fine- mne-grained metrics such as Hit@| and MRR. Having either too many too few experts is detrimental to the model's learning performance. The impact of weight À is similar. The model achieves the best results at K = 3 and À = 0.0001.
5.6 EFFICIENCY ANALYSIS (RQ4)
Even though our design introduces many new modules that have not appeared in existing methods, there is not much loss of efficiency in terms of time and space complexity. The introduction of new computational and parametric quantities occurs in two main sections, the MuJoD module and the ExID module. Our approach remains linear in the growth of complexity, and there no such level of latency as an exponential explosion. Here, present the time efficiency and GPU mem- ory usage in Table 3 Combining the SOTA MMKGC performance of MoMoK in Table and the training ef- ficiency/GPU consumption results presented in this table we can conclude that our approach achieves high efficiency and less GPU memory usage while maintaining the SOTA effect. Therefore, the time and space complexity of the present method is within reasonable limits and still performs well.
Method
Time
GPU Memory
MoMoK MMKRL OTKGE
9.8s
5900MB
7.5s
4504MB
70.1s
2540MB
MMRNS
25.5s
25582MB
Table 3: Time and GPU memory cost for different MMKGC methods.
5.7 CASE STUDY (RQ5)
To provide a more intuitive justification and interpretability for our approach, we conduct the case study from both macroscopic and microscopic viewpoints. We set a separate score for each modality in the MuJoD module and finally integrate them for joint decision-makinn the contribution each modality to the final result, we list in Table several relations where each modality score achieves the best results.
Therefore, to visualize
Notably, the relation types that each modality score S,, best predicts are diverse. These relations that perform best in the overall prediction be found in the prediction results of the different modalities. This implies that MoMoK effectively merges predictions from various modalities for joint consideration, thereby outperforming the results achieved by individual modalities.

---

Published as a conference paper at ICLR 2025
STRI
STR.2
STK33
IMG.I
IMGZ
IMG3
TXT.
IX12
TX13
(3), Staten
Figure 5: Attention weights visualization results. We select some relations and present the weights of each modality contributing to the joint representation EJoint- We further present the weights G; for K(K = 3) ReMoKEs the modality outputs ém Abbreviations for modalities: Structure (STR), Image (IMG), Text (TXT). M.k in the legend denotes the k-th expert of modality M.
Simultaneously, we delve into the micro level by analyzing the adaptive weights in MoMoK. Our design incorporates the expert decisions via series of adaptive weights in the ReMoKE, while MuJoD module also employs adaptive weights to derive the joint modality embedding from the outputs of the modalities. We select handful of relations to investigate the weights from each modality and each ReMoKE within the joint modality embedding the entities in the respective relational contexts.
As shown in Figure 5 the joint embedding of entities in varied relational contexts assigns diverse significance each modality's information. For example, PartOf attaches more weight to textual modality, while Parents relies more on image modality. Furthermore, the majority of contributions within each modality come from the same expert, and the contributions from different modalities in distinct relational contexts vary greatly. This indicates that we successfully delegate different ReMoKEs intra- auta-modality handle different relational contexts, aligning with our original intent of proposing the MoE architecture to address the MMKGC task.
CONCLUSION AND FUTURE WORK
In this paper, we propose a new MMKGC framework called MoMoK to learn modality features in diverse perspectives from the modality information of entities with relational guidance and integrate the multi-modal information through modality knowledge experts. We further decouple the expert networks and enhance the model's expressive capability through the comparative estimation of mutual information. Experimental results show that our design can achieve new SOTA results on multiple public benchmarks with both robustness, reasonability, and interpretability. Looking ahead, we can further design a more rational MoE architecture that not only accomplishes the tasks of the MMKGC but also finds ways incorporate the MMKG and the large language models realize sparse large language model with multi-modal knowledge perception.
ACKNOWLEDGMENTS
This work is founded by National Natural Science Foundation of China (NSFCU23B2055 / NS- FCU19B2027 / NSFC62306276), Zhejiang Provincial Natural Science Foundation of China (No. LQ23F020017), Yongjiang Talent Introduction Programme (2022A-238-0 and Fundamental Re- search Funds for the Central Universities (226-2023-00138), This work was supported by AntGroup.
REFERENCES
Ivana Balazevie, Carl Allen, and Timothy M. Hospedales. Tucker: Tensor factorization for knowl- edge graph completion. In EMNLP/JJCNLP (1), pp. 5184-5193, Association for Computational Linguisties, 2019.
10

---

Published as a conference paper at ICLR 2025
Shuqing Bian, Xingyu Pan, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, and Ji-Rong Wen. Multi-modal mixture of experts represetation learning for sequential recommendation In CIKM, pp. 110-119. ACM, 2023.
Antoine Bordes, Nicolas Usunier, Alberto García-Duráas Jason Weston, and Oksana Yakhnenko. Translating Embeddings for Modeling Multi-relational Data. In NIPS, pp. 2787-2795, 2013.
Zongsheng Cao, Qianqian Xu, Zhiyong Yang. Yuan He, Xiaochun Cao. and Qingming Huang. OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport. In NeurlPS, 2022.
Linlin Chao, Jianshan He, Taifeng Wang. and Wei Chu. PairRE: Knowledge Graph Embeddings via
Paired Relation Vectors. In Proc. of ACL, 2021.
Zhuo Chen, Wen Zhang, Yufeng Huang. Mingyang Chen, Yuxia Geng, Hongtao Yu, Zhen Bi, Yichi Zhang, Zhen Yao, Wenting Song. Xinliang Wu, Yi Yang, Mingyi Chen, Zhaoyang Lian, Yingying Li, Lei Cheng, and Huajun Chen. Tele-knowledge pre-training for fault analysis. In ICDE, pp. 3453-3466. IEEE, 2023a.
Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, Jiaqi Li, Xiaoze Liu, Jeff Z. Pan, Ningyu Zhang, and Huajun Chen. Knowledge graphs meet multi-modal learning: A comprehensive survey. CoRR, abs/2402.05331, 2024.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G. Learned-Miller, and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In CVPR, pp. 11828-11837 IEEE, 2023b.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. CLUB: A contrastive log-ratio upper bound of mutual information. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1779-1788. PMLR, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-IILT (I), pp. 4171-4186. Association for Computational Linguistics, 2019.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, volume 15 of JMLR Proceedings, pp. 315-323. JMLR.org, 2011.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In KDD, pp. 585-593. ACM, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825,, 2023.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Jaejun Lee, Chanyoung Chung, Hochang Lee, Sungho Jo, and Joyce Jiyoung Whang. VISTA: Visual-Textual Knowledge Graph Representation Learning. In EMNLP (Findings), pp. 7314-7328. Association for Computational Linguistics, 2023.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. Dbpedia- large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 2015.
Xinhang Li, Xiangyu Zhao, Jiaxing Xu, Yong Zhang, and Chunxiao Xing. IMF: Interactive Multi- modal Fusion Model for Link Prediction. In WWW, pp. 2572-2580. ACM, 2023.
11

---

Published as a conference paper at ICLR 2025
Ke Liang. Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang. Xiangjun Dong, and Xinwang Liu. Knowledge graph contrastive learning based on relation- symmiirrinmmmmtiiicall.l structure. IEEE Transactions Knowledge and Data Engineering. 36(1):226 238, 2023.
Ke Liang, Lingyuan Meng, Hao Li, Meng Liu, Siwei Wang. Sihang Zhou, Xinwang Liu, and Kunlun He. Mgksite: Multi-modal knowledge-dre driven site selection via intra and inter-modal graph fusion. IEEE Transactions on Multimedia, 2024a.
Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, Fuchun Sun, and Kunlun He. A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Ke Liang, Sihang Zhou, Meng Liu, Yue Liu, Wenxuan Tu, Yi Zhang, Liming Fang, Zhe Liu, and Xinwang Liu.
spatial-temporal hypergraph contrastive learning based on criminal correlations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 8733-8741, 2024c.
Ye Liu, Hui Li, Alberto Barefa-Durán, Mathias Niepert, Daniel Oñoro-Rubio, and David Rosen- blum. MMKG: multi-modal knowledge graphs. In ESWC, volume 11503 of Lecture Notes in Computer Science, pp. 459-474. Springer, 2019.
Xinyu Lu, Lifang Wang, Zejun Jiang. Shichang He, and Shizhong Liu. MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning. Appl. Intell., 52(7)7488-77497, 2022.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-expers. In KDD, pp. 1930-1939. ACM, 2018.
Haojie Pan, Yuzhou Zhang, Zepeng Zhai, Ruiji Fu, Ming Liu, Yangqiu Song, Zhongyuan Wang, and Bing Qin. Kuaipedia: a large-scale multi-modal short-video encyclopedia. CoRR, abs/2211.00733. 2022.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts, In NeurlPS, pp. 5583-8595,00 2021.
Hatem Mousselly Sergich, Teresa Botschen, Iryna Gurevych, and Stefan Roth. Multimodal Translation-Bann Based Approach for Knowledge Graph Representation Learning. In "SE@@NAAI. HLT, pp. 225 234. Association for Computational Linguistics, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey F. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expee layer. In ICLR (Poster). OpenReview.net, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: a core of semantic knowledge. In WWW, pp. 697-706. ACM, 2007.
Zhiqing Sun, Zhi-Hong Deng, Jian Yun Nie, and Jian Tang. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In ICLR (Poster). OpenReview.net, 2019.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex Embeddings for Simple Link Prediction. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pp. 207 1-2000. JMLR.org, 2016.
Denny Vrandecic and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Commun. ACM, 57(10):78-88 2014.
Meng Wang, Sen Wang. Han Yang, Zheng Zhang, Xi Chen, and Guilin Qi. Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective. In ACM Multimedia, pp. 2735-271 2743. ACM, 2021.
12

---

Published as a conference paper at ICLR 2025
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. KGAT: knowledge graph attention network for recommendation. In KDD, pp. 950-958. ACM, 2019a.
Xin Wang, Benyuan Meng, Hong Chen, Yuan Meng, Ke Lv, and Wenwu Zhu. TIVA-KG: A multimodal knowledge graph with text, image, video and audio. In ACM Multimedia, pp. 2391- 2399. ACM, 2023.
Zikang Wang, Linjing Li, Qiudan Li, and Daniel Zeng. Multimodal Data Enhanced Representation Learning for Knowledge Graphs. In IJCNN, pp. 1-8. IEEE, 2019b.
Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Image-cmbodicd Knowledge Repre- sentation Learning. In I/CAI, pp. 3140-3146. ijeai.org, 2017.
Derong Xu, Tong Xu, Shiwei Wu, Jingbo Zhou, and Enhong Chen. Relation-enhanced Negative Sampling for Multimodal Knowledge Graph Completion. In ACM Multimedia, pp. 3857-3866. ACM, 2022.
Derong Xu, Jingbo Zhou, Tong Xu, Yuan Xia, Liu, Enhong Chen, and Dejing Dou. Multimodal Biological Knowledge Graph Completion via Triple Co-Attention Mechanism. In ICDE, pp. 3928-3941 IEEE, 2023.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In ICLR (Poster), 2015.
Liang Yao, Chengsheng Mao, and Yuan Luo. KG-BERT; BERT for knowledge graph completion. CoRR, abs/1909.03399 2019.
Yichi Zhang and Wen Zhang. Knowledge graph completion with pre-trained multimodal transformer and twins negative sampling. CoRR, abs/2209.07083 2022.
Yichi Zhang, Mingyang Chen, and Wen Zhang. Modality-aware negative sampling for multi-modal knowledge graph embedding. In IJCNN, pp. 1-8. IEEE, 2023a.
Yichi Zhang, Zhuo Chen, Yin Fang, Lei Cheng, Yanxi Lu, Fangming Li, Wen Zhang, and Huajun Chen. Knowledgeable preference alignment for Iims in domain-specific question answering. CoRR, abs/2311.06503, 2023b.
Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, and Huajun Chen. Native: Multi-modal knowledge graph completion in the wild. Authorea Preprints, 2024a.
Yichi Zhang, Zhuo Chen, Lei Liang, Huajun Chen, and Wen Zhang. Unleashing the power of imbal- anced modality information for multi-modal knowledge graph completion. CoRR, abs/2402.15444, 2024b.
Yu Zhao, Xiangrui Cai, Yike Wu, Haiwei Zhang, Ying Zhang, Guoqing Zhao, and Ning Jiang. MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion. In EMNLP, pp. 10527-10536. Association for Computational Linguistics, 2022.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei Wu.
mixture of lora experts for uploadable machine learning, arXiv preprint arXiv:2406. 16989, 2024.
Ziyu Zhao Yixiao Zhou, Didi Zhu, Tao Shen, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu, and Yu Cheng. Each rank could be expert: Single-ranked mixture of experts lora for multi-task learning. arXiv preprint 250i 15103, 2025.
13

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
Multi-modd modii- Knowledge Graph Completion (MMKGC) aims to achieve missing knowledge discovery in the given knowledge graph by considering both structural information and the multi-modal content of entities. Existing MMKGC methods can be divided into three categories based on the design ideas:
Modality Fusion Methods. These methods integrate multi-modal embeddings of entities with their structural embeddings for triple plausibility estimation. Early approaches like IKRL (Xie et al., 2017). TBKGC (Sergich et al. 2018), TransAE (Wang et al. 2019b) employ multiple translation-based score functions for embeddings from diverse modalities. Later approaches such OTKGE (Cao al.) 2022). VISTA (Lee al., 2023). and AdaMF (Zhang et al. 2024b) introduced more sophisticated feature fusion techniques such as optimal transfer, transformer, and adversarial training into multi-modal fusion.
Modality Ensemble Methods. These methods train separate models for each modality and combine them for a joint final decision. MoSE (Zhao et al. (0022) design three different ensemble methods adaptively combine the single-modality models. IMF (Li al. 2023
designs an interactive module among modalities with both fusion and ensemble.
Modality-awar Negative Sampling Methods. These methods involve the negative sam- pling design by considering the extra multi-modal information for better contrastive learning during the MMKGC model training process. For example, MANS (Zhang al.) 2023a) designs visual negative sampling strategies to align the visual modality with triple structure. MMRNS (Xu et al. 2022) introduces relation-enhanced negative sampling method by considering relation- -uided negative sample generation.
In this paper, our research focuses on designing a mixture-of-expprrs framework with ensemble learning for MMKGC, which would be the combination of the modality fusion and modality ensemble methods.
B MODEL DESIGN AND ANALYSIS
In this section, we present a detailed theoretical analysis of the CLUB loss Cheng et (2020) we used in Section 4.3
Theorem 1 For two entity embedding i Vmnj (simplified as V4 and V, in the following derivation for the sake of typography), they can be treated as two random variables. The upper bound of their mutual information I(V;; Vi) is:
ICLUB(24 V2) = E.D..,) Dogp(V) VJ] - Blgg P(V; [4)]
(11)
This theorem can be proven with the following derivation. We calculate the difference between ICLUD (V:; V2) and I(V;; V2)) by:
Proof 1
A -ICLUB (M;; V,) - I(V;; 1/y)
(lw,,,,,looPP() V.)] Ber,) [V)] - Boug,,, V3) log PVV))
Epp(,,,, (loggoog))))))))) EE(,, (lgg)))))())) [2)]
EEpv,) (VV)) (pcv)(lop(()) [J]]]
(12)
Besides, according to Jensen inequality, have:
log (VV)) (Ep (PV) V.)) 2 Epo,)loog P(V VA] A TCLUB( (V;;; V1) T(>;; V3y
(13)
Therefore, ICLUB (V;; V,) is an upper bound of 1(V;; V2). Besides, the real conditional distribution P(V; V.) is hard measure. Therefore, variational distribution (V, V.) parameterized by neural network 0 is usually employed to approximate the real distribution. Therefore, the variational CLUB objective be denoted as:
VoLUB(V:3 (V: (, log V.)] (v.) (V. Dlo qo (V) V.)]
(14)
14

---

Published as a conference paper at ICLR 2025
It can be proved that with good variational network go. I' can still hold an upper bound of the mutual information. Here we denote qo (V,, V2) = qo V))(()) and introduce such a new theorem:
Theorem 2 If we have
KLpp va (V;, V2)) s (L) V;))
(15)
then we have I(V;; V2) & CCCLBB(2 V)). KL represents the Kullback-Leibb divergence between two probability distributions.
Proof 2 We can have the following derivation:
Á = I'c,un V2) - I(V;; V,)
Ep(V.,,,) [og go(); V:]] - Ep(,.) Ep(, fog to (V; [.)] Re V3) 1o3) [Epc,,) [(V;]] Ep(()pces)) 2o (V) V:)]
- (eMe.,))logg(( r(V; V:]] Bp(r, log (0()) V:]]].
p(V;)
p(V, V:)
=E(c))ss)))oogg (o VzJ -E 1, [log go(V; V:j
EpCV))cC, (o VV)((),) E Bp(M,,,) flo p(V; Va)p(V;) go (V V))(()))
p(V))((V,)
(T. Vy)) - KT(p(V,, V2) (go V2))
(16) Therefore, L'LUE (v; ; V2) is an upper bound of 1(V;; V2) if and only if K Lpp)), V;) v2)) S (V,) (V,, V2)). The equality holds when V., V/ are independent.
This theorem indicates that ICCLUR (V;; V2) can still be an upper bound for I(V; Vj) if the varia- tional distribution qo (V.. V2) is closer to p(V,, Vj) than to p()))p(V,). Hence, we can minimizing KL,,(V,, (e)))lg/,,,, V2)) to achieve this goal.
In our specific implementations, we employ several neural networks Qa.m to estimate the variational distribution. We further design two loss (Letub in Equation X and Leaid in Equation 9 to implement the CLUB module. Letub is designed to estimate the mutual information among different MoKEs and Levid is designed to optimize the neural networks Qa,m- These two losses correspond to Theorem and Theorem 2 derived earlier.
C EXPERIMENT DETAILS
C.1 BASELINES
The KGC baselines we compared in our experiments can be divided into three categories:
(1). Uni-modal KGC methods: TransE [Bordes et al. 2013), DistMult (Yang et al. 2015). ComplEx
(Trouiloon et al. 2016). RotatE (Sun et al. 2019), PairRl (Chao et al. 2021). and TuckER (Balazevic et al.) 2019). These methods only consider the triple structural information in their designs and learn structural embeddings for KGC.
(2). Multi-modal KGC methods: IKRL Xic et al.
2017). 7) TBKGC (Sergich et al. 2018), TransAE
WWag et al. (019b)), RSME (Wang et al.) 2021). 1). MMKRL (Lu al. 2022D. VBKGC ZZang &
Zhang 2022). OTKGE (Cao et al. 2022). MoSE (Zhao et al. 2022). y MMRNS (Xu et al.) 2022). MANS (Zhang et al. 2023a), IMF (Li et al. 2023), QEB (Wang et al. 2023), VISTA (Lee et al. 2023). and AdaMF (Zhang et al.) 2024b). These methods employ mudti-modal information such as texts and images to enhance the entity representation learning, achieving better KGC performance.
(3). Negative sampling methods: MANS (Zhang et al. (023a) and MMRNS (Xu et al. (022).
These methods attempt to modify the traditional negative sampling methods with the multi-moda information from entities to make fine-grained contrast during training.
In the experiments, we conduct a comprehensive comparison with the mentioned 20 baselines to demonstrate that MoMoK can leam better entity representations to enhance the MMKGC process. Some methods (Yao et al. 2019) fine-tun tening the pre-trained models are orthogonal to our design philosophy and paradigm do not compare with them.
15

---

Published as a conference paper at ICLR 2025
Table 5: Statistical information of the four MMKGs in experiments. The image and text modality
features are provided by the original datasets and kept the same for all baselines.
Image
Text
Dataset
#Entity
#Relation
#Train
#Valid
#Test
Num
Dim
Num
Dim
MKG-W (Xu et al.
2022
15000
169
34196
4276
4274
14463
383
14123
384
MKG-Y Auelal 2122
15000
28
21310
2665
2663
14244
383
12305
384
DBISK
12842
279
79222
9902
9904
12818
4096
9078
768
KVC16K
Zhang et al 2024a
16015
4
180190
22523
22525
14822
768
14822
768
Table Full results on the MKG-W and MKG-Y datasets.
MKG-W
MKG-Y
Method
MRR
Hitl
Hit3
Hit10
MRR
Hit1
Hit3
Hit10
TransE
29.19
21.06
33,20
44.23
30.73
23.45
35.18
43.37
DistMult
20.99
15.93
22.28
30.86
25.04
19.33
27.80
35.95
ComplEx
24.93
19.09
26.69
36.73
28.71
22.26
32.12
40.93
RotatE
33.67
26.80
36.68
46.73
34.95
29.10
38.35
45.30
IKRL
32.36
26.11
34.75
44.07
33.22
30.37
34.28
38.26
TBKGC
31.48
25.31
33.98
43.24
33.99
30.47
35.27
40.07
TransAE
30.00
21.23
34.91
44.72
28.10
25.31
29.10
33.03
MMKRL
30.10
22.16
34.09
44.69
36.81
31.66
39.79
45.31
RSME
29.23
23.36
31.97
40.43
34.44
31.78
36.07
39.09
VBKGC
30.61
24.91
33.01
40.88
37.04
33.76
38.75
42.30
OTKGE
34.36
28.85
36.25
44.88
35.51
31.97
37.18
41.38
MoSE
33.34
27.78
33.94
41.06
36.28
33.64
37.47
40.81
IMF
34.50
28.77
36.62
45.44
35.79
32.95
37.14
40.63
QEB
32.38
25.47
35.06
45.32
34.37
29.49
36.95
42.32
VISTA
32.91
26.12
35.38
45.61
30.45
24.87
32.39
41.53
MANS
30.88
24.89
33.63
41.78
29.03
25.25
31.35
34.49
MMRNS
35.03
28.59
37.49
47.47
35.93
30.53
39.07
45,47
MoMoK
35.89
30.38
37.54
46.13
37.91
35.09
39.20
43.20
C.2 EVALUATION PROTOCOLS
We conduct link prediction task the datasets, the mainstream MMKGC task. Following existing works, we use rank-based metrics like mean reciprocal rank (MRR) and Hit@ (K=1, 3, 10) to evaluate the results. Besides, employ the filter setting in the prediction results remove the candidate triples that already appeared in the training data for fair comparisons. The final results are the average of both head prediction and tail prediction. MRR and @it@l K can be calculated as:
Tread
MRR
(17)
Thú
VzAA
i=1
Thaae
Hit@K
Σ
(1(r S K) 1(r I K))
(18)
(Teesel
where Tha and rt, are the results of head prediction and tail prediction respectively, Trast is the test triple set.
C.3 ADDITIONAL EXPERIMENTAL RESULTS
We present the additional experimental results in this section.
C.3.1 FULL RESULTS ON MKG-W AND MKG-Y
In our experiments, we present the MRR and Hit@ results in Table T We now present the results for the complete set of four metrics in both datasets in the Table
16

---

Published as a conference paper at ICLR 2025
Table 7: MMKGC results the FB15K-237 dataset.
Method
MRR
@it@
Hit@3
Hit@ 10
IKRL*
19.4
28.4
45.8
TransAE*
19.9
31.7
46.3
RSME*
24.2
34.4
46.7
MoSE-AI#
25.5
37.6
51.8
MoSE-BI#
28.1
41.1
56.5
MOSE-MI
26.8
39.4
54.0
QEB
29.93
20.73
33.37
48,03
AdaMF
32.56
23.33
35.85
51.13
MoMoK (Ours)
36.08
27.54
39.60
55.73
Table 8: Ablation study the MKG-Y and KVC16K datasets. The results with are from the MoSE paper (Zhao et al. 2022). means the results are not reported in the given paper.
MKG-Y
KVC16K
Setting
MRR
Hit@1 1
Hit@3
Hit 10
MRR
Hit@
Hit@3
Hit@ 10
Full Model
37.91
35.09
39.20
43.20
16.87
10.53
18.26
29.20
(1.1) Structural Modality
35.66
32.14
37.61
41.51
15.66
9.52
16.95
27.54
(1.2) Image Modality
35.99
32.84
37.98
41.23
15.50
9.56
16.59
27.11
(1.3) Text Modality
35.73
32.41
37.76
41.18
15.56
9.61
16.68
27.16
(1.4) Joint Modality
35.83
32.14
37.81
42.60
16.45
10.22
17.61
28.59
(2.1) relational temperature
37.17
33,93
38
42.85
16.74
10.31
18.16
(2.2) w/o noise 8...
37.63
34.62
43.11
16.77
10.40
18.08
29.07
(2.3) w/o adaptive fussoo
36.54
33.05
38.17
42.28
16.55
10.28
17.69
28.70
(2.4) w/o joint training
35.33
31.69
37.01
40.99
15.92
9.69
17.20
27.91
(2.5) w/o ExID
36.37
32.56
38.64
42.70
16.05
9.87
17.24
28.05
These added experimental results allow us to draw further conclusions. For example, we can find that our method MoMoK outperforms baselines on MRR/Hit @ 1/Hit on MKG-W and Hit@ @ on MKG-Y. Y. We can conclude that our method will perform better on dine-graine ranking metrics such Hit@ I than the yoarse-grained ranking metric Hit @ 10. The trend reflected in the new results is the same as DB15K and KVC16K.
C.3.2 ADDITIONAL RESULTS ON F515K2237
We present more results FB15K-237, shown in the following Table These results indicate that MoMoK still performs well on classic MMKG benchmarks like FB15K-237. Only MoSE-BI currently slightly better. Compared with recent baselines like AdaMF, MoMoK performs better. Based on our observation of the source code, MoSE achieves this result by setting the dimension of embedding to 2000, while our method MoMoK uses 250.
Compared with MoSE, MoMoK is an end-to-end training framework, MoSE needs to learn the parameters of the ensemble once more after the model trained, which is also feature of our design.
C.3.3 ADDITIONAL. ABLATION STUDIES
We present the experimental results of MKG-Y and KVC16K in the Table do more in-depth analysis of the whole ablation study.
These experimental results are consistent with those shown in the paper for MKG-W and DB15K. That is, each module contributes to the final performance. Besides, we can also some analysis from the new experimental results on this issue of the small impact of the different modules that you mentioned. On the one hand, we conducted a fine graigeaine ablation study, with specialized experiments several key designs. On the other hand, contribution of the same module in different datasets. This is related to the dataset itself, in addition to the fact that the foundational elements of our entire framework, such the TuckER score itself,
observe that there is also big difference in the
17

---

Published as a conference paper at ICLR 2025
have better performance and subsequent designs have made significant enhancements based on this foundation.
D LIMITATIONS OF OUR WORK
Our main work is the design and implementation of a novel MMKGC framework. Of course, there some limitations to our work. The main points are as follows:
Limitations of task scenarios. Our research focuses on specific research field called MMKGC and our method MoMoK designed specifically for this task. We do not generalize this framework to more multi-modal tasks.
Integration with LLM trends. Our approach uses classical embedding- based approach in studying the MMKGC problem and does combine the MMKG with the latest LLM trends in synergistic way.
Limitations the experiment. Due to the lack of standard datasets for super large-scale experiments MMKG, experiments were conducted mainly on medium-sized datasets.
E BROADER IMPACTS OF OUR WORK
Our work focuses on reasoning about multi-modal knowledge, which can help discover new possible associations in large-scale semantic networks and encyclopedic knowledge and expand existing encyclopedic knowledge bases such as Wikidata, etc. The positive social impact of our work is to help build and expand the Internet knowledge-sharing community, and to make more accumulation and deposition of linked data. We do not believe that our research will have negative social impact, and we will also take active steps to avoid misuse of our methods.
18


Reviews:
[{"review_id": "nEGNmEUNWW", "reviewer": "Reviewer_VfNT", "summary": "This paper tackles MKGE problems by proposing a Mixture of Modality Knowledge Experts framework. The authors develop relation-guided modality knowledge experts to obtain relation-aware modality embeddings while reducing mutual information among the experts. Empirical studies validate the model's effectiveness.", "questions": "please see above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Experiments in complex environments demonstrate MoMoK's resilience to noise, data sparsity, and missing modalities.  \n2. This work introduces a novel expert information disentanglement (ExID) module to separate expert decisions for each modality.  \n3. The rationale for designing the relation-aware MKGC method is evident.", "weaknesses": "1. Metrics Reporting: Table 2 displays only the Hit@1 and MRR metrics for MKG-W and MKG-Y, showing modest performance improvements for MoMoK. Please include additional metrics for MoMoK, AdaMF, and MMRNS to better assess the reported gains.\n\n2. MoMoK on Classic MMKGC Datasets: Why wasn't MoMoK evaluated on classic MMKGC datasets like WN9-IMG, FB-IMG, WN18-IMG, and FB15K-237-IMG?\n\n3. Modal Data and Encoder Details: The notation $\\mathcal{X}_m(e)$ indicates information from multiple modalities, yet four datasets are used. Please clarify the specific modalities for each dataset, how multiple images are handled, and whether VGG and BERT are consistently used as encoders.\n\n4. Notation Standardization: Some formulas, such as $\\mathcal{E}_h$ in formula (1), could benefit from standardized notation to improve clarity and consistency.\n\n5. Relation-Aware Temperature ($\\mathcal{E}_r$): The relation-aware temperature $\\mathcal{E}_r$ is important for GFN computation, but its definition is absent. Please provide its formal definition and its impact on model performance.\n\n6. Score Function Comparison: The proposed method employs Tucker decomposition as its score function, while the baseline methods utilize various other functions.\n\n7. Ablation Study: Ablation experiments indicate that components like adaptive fusion, noise $\\delta_{r}$, and relational $\\epsilon_{m}$ may have minimal individual impacts. Further analysis or emphasis on their significance in different experimental contexts is needed.\n\n8. Ablation Experiments for MKG-Y and KVC16K: Can MKG-Y and KVC16K datasets be included in the ablation experiments? This would enhance the assessment of the method's performance across diverse data and conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d6QR2TeXTf", "reviewer": "Reviewer_mPTp", "summary": "This paper proposes a novel framework, Mixture of Modality Knowledge Experts (MoMoK), which aims to improve the quality of multimodal entity representation learning through relation guidance and expert information decoupling in multimodal knowledge graphs (MMKGs). MoMoK achieves better triple prediction by designing a relation-guided modal knowledge expert network, adaptively aggregating multi-view embeddings, and adopting a multimodal joint decision mechanism. Experimental results show that MoMoK performs well on four public MMKG benchmarks and significantly outperforms existing methods.", "questions": "1. The parameter \\lambda seems small enough. Can you explain why club loss can improve the model by about one point in almost every metric under a weight of 1e-4 (Table 3)?\n2. The method seems to be insufficiently integrated with the multimodal large model. This paper uses BERT and VGG to extract the original features of the model. Have you tried to use a multimodal large model to represent them uniformly? I would like to know the performance comparison between the embedding representation of the multimodal large model and MoMoK.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper considers the modal information hidden in different relational contexts and designs a relation-guided modal knowledge expert network to adaptively aggregate multi-view embeddings.\n2. On four public MMKG benchmarks, MoMoK significantly outperforms recent baseline methods and demonstrates its superior performance in complex scenarios.\n3. Through detailed ablation analysis, this paper verifies the effectiveness of the design and provides intuitive case studies to enhance the interpretability of the model.", "weaknesses": "1. Table 2 lacks a comparison of the number of model parameters. Although the authors have discussed the training memory consumption and time efficiency of some methods in Section 5.6, it is beneficial to supplement these values for all baseline models in Table 2 for fair comparison.\n2. The description in Table 4 does not seem to be that comprehensive. Does the GPU memory here refer to the memory during inference or the memory during training? Does the time in seconds represent the time it takes to infer one sample?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QJfazw7FD5", "reviewer": "Reviewer_Diph", "summary": "This paper proposes a multi-modal mixture-of-experts entity representation framework MoMoK for knowledge graph completion task. MoMoK consists of three different modules called relation-guided modality knowledge experts, multi-modal joint decision, and expert information disentanglement.", "questions": "- Please make a response to the comments in weaknesses.\n- What exactly are the advantages of the traditional embedding-based approach over the LLM-based approach for knowledge graph completion?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Multi-modal knowledge graph representation learning is a hot topic in the KG community.\n- The three modules in the design of the method proposed by the authors are self-consistent and the authors provide a more detailed theoretical proof.\n- The experimental part of the workload is sufficient, and the authors have done more experiments to prove the effectiveness, robustness, and efficiency of the method. Detailed case study and visualization results are also provided.", "weaknesses": "- The discussion of MAIN RESULTS needs to go further, have the authors explored in depth why MKG-Y has a lesser MRR than baseline AdaMF?\n- The fonts of the figure3 and figure4 are not consistent.\n- The detailed settings of Section5.4 should be more carefully introduced. About how the training datasets are built and the detailed evaluation protocol.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RDg0ANXLEk", "reviewer": "Reviewer_rUdt", "summary": "This paper introduces a novel model named MOMOK (Mixture of Modality Knowledge Experts), designed to address the challenges in multimodal knowledge graph completion tasks. The innovation lies in the relation-guided multimodal expert networks, which dynamically adjust the weighting of modal information based on relation, enhancing the model's completion performance. The experimental results demonstrate that MOMOK surpasses existing baseline models across multiple public datasets, showcasing its effectiveness in handling multimodal information. The paper is well-writing with contributions in both theoretical design and empirical results.", "questions": "1. Given that the expert networks rely on raw embeddings, how does the model ensure that sufficient diversity within each modality is captured? Are there mechanisms in place to mitigate the potential loss of key details due to the use of compressed embeddings?\n2. Could you provide further clarification on the rationale behind using the conditional probabilities of different entities for disentanglement in formula 8? How does this approach ensure effective disentanglement between experts?\n3. In the final case study, the authors visualize the weighting between modalities. How does the model distinguish and prioritize features within a single modality?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of relation-guided multimodal expert networks with adaptive weighting mechanisms enables the model to fully exploit different perspectives within each modality based on relational context, which represents a novel and impactful focus.\n2. By minimizing mutual information, the model effectively reduces redundant information between modalities, enhancing disentanglement among the experts. This is a particularly noteworthy design feature.\n3. The ablation experiments validate the importance of each module in the model's performance, particularly highlighting the significance of joint training and information disentanglement, proving the rationale behind the design.\n4. The paper presents five well-defined research questions (RQs) that thoroughly address the model’s strengths, performance in complex environments, and the contribution of each design module. This makes the experimental section well-structured and logically cohesive.", "weaknesses": "1. The novelty is somewhat limited. The paper claims existing works lack considering the relational context, and proposes MoMoK frameworks by incorporates relation-guided modality knowledge experts for each modality, which constructs expert networks in each modality. But in my opinion, there are also lots of prior work (such as MoSE) considering relational context. \n2. It would be beneficial to introduce datasets used in baselines for comprehensive comparison. The baselines, such as MoSE, further conduct experiments on (FB-15K, WN18, WN9) which this paper did not used. Including these datasets would be helpful to further demonstrate the robustness of the proposed model.\n3. The expert networks rely on raw embeddings, which may limit their ability to fully capture the diversity within each modality. This could lead to less distinct perspectives being generated by the experts, especially if the raw embeddings have already compressed or lost key details, thereby reducing the overall effectiveness of the expert network design.\n4. While the paper introduces CLUB to estimate the upper bound of mutual information and designs a corresponding loss function, it lacks sufficient explanation for why the conditional probabilities of different entities are used for disentanglement in formula 8.\n\n\nOther Minor Weaknesses:\n\n1. The related work section is too brief and fails to emphasize the superiority of the proposed framework compared to existing approaches. For example, in the experimental section, you mentioned the similarities between your method and MoSE and IMF as baselines, but you did not provide a more detailed discussion of the differences. This could be elaborated in the related work section to clarify the distinctions between your approach and these existing methods.\n2. The paper discusses the use of three modalities, image, text, and structure\nbut it appears to lack a clear explanation regarding how the raw embeddings for the structural modality are obtained. Neither the main body of the paper nor the appendix provides detailed information about the specific model or method used to generate raw embeddings for the structural modality.\n\nTypos:\n\n1. Page2 'constrative' maybe is 'contrastive'\n2. The second-best method in the MRR indicator of the MKG-W dataset in Table 2 should be MMRNS (35.03), but it was incorrectly marked as IMF (34.50).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f4mBukcqG9", "reviewer": "Reviewer_mwpc", "summary": "The authors argue the limitations of entity-wise embedding fusion in existing MMKGC work and introduce a new method, MoMoK, which can learn relation-guided entity representations for each modality. The experimental results demonstrate the superiority and robustness of MoMoK.", "questions": "1. In the second paragraph of Section 4.1, are $\\mathcal{W}\\_{m,1}$, $\\mathcal{W}\\_{m,2}$, ... learnable matrics? What are their shapes? Also, what is the exact operation in $\\mathcal{V}\\_{m,i}^{e}=\\mathcal{W}\\_{m,i}(e\\_m)$? If it is just a simple multiplication, I feel using parentheses is improper. \n2. In Eq. (2), what is the purpose/function of the temperature $\\varepsilon_r$?\n3. In Eq. (3), I think the relation $r$ is missing in the components $P\\_m(\\hat{e}\\_{m})$ and $P\\_n(\\hat{e}\\_{n})$?\n4. In line 266, where the joint score of each triple is treated as the sum of the scores across all modalities, did the authors try other designs, such as using only the '$J$' modality score? I am curious about the results of using only the '$J$' modality.\n5. What's the setting of the number of experts (the value of $K$) in the main experiment? This should be described in the implementation details.\n6. In Section 5.1, what is the difference between the MKG-W and MKG-Y datasets?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written, and I was able to grasp authors' motivation and the general process of MoMoK at first glance. The algorithm is clearly presented overall. The performance of MoMoK surpassed existing methods by a significant margin. Sufficient experimental evidence supports the robustness of MoMoK.", "weaknesses": "I think **the discussion of the importance of KG Completion task, especially its position in the context of the pervasiveness of LLMs today, is lacking**. This reduces the overall interest and significance of the work from the audience's perspective, considering LLMs' strong performance in general knowledge reasoning. I suggest the authors add a brief discussion in the introduction or in the last section. Additionally, several places lack detail and need clarification (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)