Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning

## Abstract

Abstract Learning high-quality multi-modal entity representations is an important goal of multi-modal knowledge graph (MMKG) representation learning, which can enhance reasoning tasks within the MMKGs, such as MMKG completion (MMKGC). The main challenge is to collaboratively model the structural information concealed in massive triples and the multi-modal features of the entities. Existing methods focus on crafting elegant entity-wise multi-modal fusion strategies, yet they overlook the utilization of multi-perspective features concealed within the modalities under diverse relational contexts. To address this issue, we introduce a novel framework with M ixture o f Mo dality K nowledge experts ( MoMoK for short) to learn adaptive multi-modal entity representations for better MMKGC. We design relation-guided modality knowledge experts to acquire relation-aware modality embeddings and integrate the predictions from multi-modalities to achieve joint decisions. Additionally, we disentangle the experts by minimizing their mutual information. Experiments on four public MMKG benchmarks demonstrate the outstanding performance of MoMoK under complex scenarios. Our code and data are available at https://github.com/zjukg/MoMoK .

## 1 Introduction

Multi-modal knowledge graphs (MMKGs) (Chen et al., ) are an extension of traditional knowledge graphs (KGs) (Liang et al., ), encompassing rich modality information such as images and textual descriptions of large-scale entities, which bridges structured knowledge triple and unstructured multi-modal content together. Based on such a specialized and expressive data organization format, MMKGs have evolved the emerging infrastructure of Artificial Intelligence (AI), contributing to numerous AI-related fields like large language models (Zhang et al., ), recommendation systems (Wang et al., ), and other practical applications (Chen et al., ).

Learning better entity representation is a crucial topic in the representation learning and KG community. It aims to encode the complex information in the given KG to perform KG reasoning tasks like knowledge graph completion (KGC), which is an interesting and important task seeking to automatically discover new knowledge from the existing KGs. KGC allows for the missing entity prediction to a given entity-relation query, e.g., (ICLR 2025, Located In, ?). Traditional KGC usually learn entity representations by modeling the triple structure in the embedding space. As for MMKG, the situation becomes more complex. Multi-modal knowledge graph completion (MMKGC) further enhances the entity embeddings with multi-modal features, aiming to collaboratively model the triple structure and multi-modal content to achieve robust prediction.

Existing MMKGC methods (Wang et al., ; Xu et al., ) typically employ a multi-modal fusion module to integrate the information from different modalities to obtain joint entity embeddings. These entity embeddings are then mapped into a scalar score along with the relation embeddings as a basis for assessing the triple plausibility. MMKGC, being a prediction task in a multi-relational scenario, is influenced by different relational contexts, which in turn affect the selection and utilization of entity modality features. As illustrated in Figure 1, different sections of varied modality information emphasize their respective significance when making predictions based on different relationships. However, such a conventional paradigm overlooks the information diversity both inter-modality and intra-modality. Different modalities can represent various aspects of entity information, and information within the same modality can also play different roles depending on the relational context. If vanilla multi-modal fusion is performed directly at the entity level without considering the relational context, it can result in low utilization of this multi-modal information and finally learn immutable entity embeddings across different relational contexts, thereby limiting the model’s performance. This limitation is particularly pronounced in realistic scenarios where many entities are subject to modal noise and incomplete information, which can further challenge the model’s ability to utilize modal information effectively.

Figure: Figure 1: Different relational context requires different modality information for proper prediction.
Refer to caption: x1.png

To address these issues, we propose a Mixture of Modality Knowledge experts (MoMoK) framework in this paper. MoMoK incorporates relation-guided modality knowledge experts for each modality, which constructs expert networks in each modality. These expert networks, guided by the relational context of the current triple, adaptively aggregate the multi-view embeddings for entities. Further, MoMoK employs multi-modal joint decision to integrate the modality embeddings as a new joint modality and achieve comprehensive triple prediction in an ensemble manner. Ultimately, we employ an expert information disentanglement module to differentiate learning across different expert networks with constrative mutual information estimation, aiming to force different experts to specialize in different relational contexts. This entire process can be likened to each modality functioning as a senior expert, gathering the insights of junior experts within the corresponding modality, based on the principle that multiple heads are better than one. These insights are then communicated and integrated across modalities to facilitate more comprehensive decision-making. We conduct comprehensive experiments on four public MMKG benchmarks to demonstrate the effectiveness of our framework MoMoK with further exploration to validate its properties. Our contribution can be summarized as:

- •
We address the problems in modality information utilization by MMKGC models and propose MoMoK with relational-guided modality experts and multi-modal joint decision to learn better entity representations and unleash the power of multi-modal information in MMKGs.
- •
We examine the learning of different modal experts through the lens of mutual information estimation, and propose to decouple and discretize the expert information within a modality using mutual information comparison estimation. We present detailed theoretical analysis to justify the design.
- •
We conduct extensive experiments against 20 recent baselines on four MMKG benchmarks to demonstrate the state-of-the-art (SOTA) performance of MoMoK and further explore its robustness, reasonability, and interpretability in the complex environments.

## 2 Related Works

#### Multi-modal Knowledge Graph Completion (MMKGC)

MMKGC (Chen et al., ) aims to automatically discover new knowledge triples from the existing MMKGs by collaboratively modeling the triple structure and multi-modal information (e.g., images and textual descriptions) in the MMKGs. Mainstream MMKGC methods (Xie et al., ) explore multi-modal fusion in the same representation space to measure the triple plausibility from multi-views. Advanced multi-modal fusion techniques such as optimal transport (Cao et al., ), modality ensemble (Li et al., ), self-attention (Chen et al., ), and adversarial training (Zhang et al., ) have been continuously introduced into MMKGC. Some work (Zhang et al., ) also tries to improve the negative sampling process commonly used in KGC.

#### Mixture-of-Experts (MoE)

MoE is a special model ensemble and combination method that is widely used in AI-related fields like computer vision (Chen et al., ; Riquelme et al., ), natural language processing (Jiang et al., ), recommendation (Ma et al., ; Bian et al., ; Hou et al., ), and so on. MoE usually divides a given task into multiple subtasks to solve them with individual expert models and design a routing module to select suitable experts to solve the current task. The MoE architecture creates a buzz due to its successful use in large language models (LLMs) (Jiang et al., ) because it can efficiently train larger and stronger LLMs.

## 3 Problem Definition

A general KG can be formalized as $\mathcal{KG}=(\mathcal{E},\mathcal{R},\mathcal{T})$ where $\mathcal{E},\mathcal{R}$ are the entity set, the relation set respectively. $\mathcal{T}=\{(h,r,t)\mid h,t\in\mathcal{E},r\in\mathcal{R}\}$ is the triple set. Furthermore, MMKGs have a modality set denoted as $\mathcal{M}$, encapsulating different modalities in the MMKGs. For an entity $e\in\mathcal{E}$, its modality information of modality $m\in\mathcal{M}$ is denoted as $\mathcal{X}_{m}(e)$. For different modalities, the elements in it have different forms. For instance, $\mathcal{X}_{m}(e)$ can be a set of images for image modality and some video clips for video modality. Note that the triple structure ($S$) is also an extra modality where the structural information is embodied in the triple set $\mathcal{T}$.

MMKGC designs a score function $\mathcal{S}(h,r,t):\mathcal{E}\times\mathcal{R}\times\mathcal{E}\rightarrow
\mathbb{R}$ to discriminate the plausibility of a given triple $(h,r,t)$. In this context, a higher score implies a more plausible triple. Entities and relations are embedded into continuous vector spaces for data-driven learning. For MMKGs, all the modality information of each entity will be represented as modality embeddings $\bm{e}_{m}(m\in\mathcal{M})$ to participate in the triple score calculation by multi-modal fusion and integration. During training, negative sampling (NS) is widely used to construct manual negative triples for contrastive learning because KGs only have observed positive triples. The negative triples are denoted as:

$$ $\mathcal{T}^{\prime}=\{(h^{\prime},r,t)\mid(h,r,t)\in\mathcal{T}\cap h^{\prime }\in\mathcal{E}\setminus\{h\}\}\cup\{(h,r,t^{\prime})\mid(h,r,t)\in\mathcal{T} \cap t^{\prime}\in\mathcal{E}\setminus\{t\}\}$ (1) $$

which is generated by a random replacement of entities in the positive triple. During inference, the MMKGC model is usually evaluated with the link prediction task (Bordes et al., ) to predict the missing head or tail entity in the given query $(?,r,t)$ or $(h,r,?)$. For each candidate $e\in\mathcal{E}$, the score of the triple $(h,r,e)$ or $(e,r,t)$ is calculated and then ranked across the entire candidate set.

Figure: Figure 2: Overview of our proposed MoMoK framework, which consists of three core components: the relation-guided modality knowledge experts (ReMoKE), multi-modal joint decision (MuJoD), and expert information disentanglement (ExID). ReMoKE constructs a mixture-of-experts network in each modality under the relational context. MuJoD aims to achieve multi-modal fusion and collaborative prediction by integrating the decision from multi-modalities. ExID further enhances the ReMoKEs by minimizing their mutual information.
Refer to caption: x2.png

## 4 Methodology

In this section, we will present our proposed framework called Mix of Modality Knowledge experts (MoMoK for short) to achieve robust MMKGC. There are three key components to our design: relation-guided modality knowledge experts (ReMoKE for short), multi-modal joint decision (MuJoD for short), and expert information disentanglement (ExID for short).

### 4.1 Relation-guided Modality Knowledge Experts

To better learn the embedding of different perspectives intra-modalities, we introduce a module called relation-guided modality knowledge experts (ReMoKE) to build expert networks in each modality. First, for each modality $m\in\mathcal{M}$, the entity $e\in\mathcal{E}$ possesses a raw modality feature $e_{m}$, derived from the modality data $\mathcal{X}_{m}(e)$. For image and text modality, a pre-trained model like VGG (Simonyan & Zisserman, ) and BERT (Devlin et al., ) would be employed to extract the raw modality feature. As for the structure modality, the raw modality feature will be learned from scratch with the triple data during training.

We then learn the multi-pespective embeddings $\mathcal{V}^{e}_{m,1},\mathcal{V}^{e}_{m,2},\cdots,\mathcal{V}^{e}_{m,K}$ of the entity $e$ and modality $m$ by establishing $K$ modality knowledge experts (MoKE) for each modality denoted as $\mathcal{W}_{m,1},\mathcal{W}_{m,2},\cdots,\mathcal{W}_{m,K}$. This process can be represented as $\mathcal{V}^{e}_{m,i}=\mathcal{W}_{m,i}(e_{m})$. Then we design a relation-guided gated fusion network (GFN) to facilitate intra-modality entity embedding fusion with relation guidance. The output entity embedding for modality $m$ and relation $r$ is denoted as $\widehat{\bm{e}}_{m,r}=\sum_{i=1}^{K}G_{i}(\mathcal{V}^{e}_{m,i},r)\mathcal{V}
^{e}_{m,i}$ where ${G}_{i}$ is the weight for each MoKE calculated by the GFN:

$$ $G_{i}(\mathcal{V}^{e}_{m,i},r)=\frac{\exp\left((\mathcal{U}_{m}(\mathcal{V}^{e }_{m,i})+\delta_{m,i})/\sigma(\varepsilon_{r})\right)}{\sum_{j=1}^{K}\exp\left ((\mathcal{U}_{m}(\mathcal{V}^{e}_{m,j})+\delta_{m,j})/\sigma(\varepsilon_{r}) \right)},\quad\mathrm{where}\quad\delta_{m,i}\sim\mathcal{N}(0,\mathcal{U}_{m} ^{\prime}(\mathcal{V}^{e}_{m,i}))$ (2) $$

where $\mathcal{U}_{m},\mathcal{U}^{\prime}_{m}$ are two projection layers and $\delta_{m,i}$ is tunable Gaussian (Shazeer et al., ) noise to balance the weights for each MoKE and augment the robustness of the MMKGC model. This is a design (Shazeer et al., ) that has been proven to work. Besides, we add a learnable relation-aware temperature $\varepsilon_{r}$ with a sigmoid function $\sigma$ to limit the temperature in the range $(0,1)$. We aim to acquire an entity modality embedding within the relational context of the current prediction before making the final decision. This approach allows us to introduce the relational context to guide the modality embedding learning in MoKEs, thereby enabling our MoKEs to extract relation-aware modality embeddings. Besides, the MoKEs will be differentiated to adapt to different relational contexts with the design of GFN. We can learn dynamical modality embeddings of entities that change in different relational contexts.

### 4.2 Multi-modal Joint Decision

With the ReMoKE module, we can obtain relation-guided modality embeddings $\widehat{\bm{e}}_{m,r}(m\in\mathcal{M})$ for each entity under relational context. Subsequently, we equip the model with the ability to amalgamate information from various modalities to facilitate joint decision-making via MuJoD module. MuJoD first accomplishes multi-modal entity embedding fusion by learning a group of adaptive weights for each entity as:

$$ $\widehat{\bm{e}}_{Joint,r}=\frac{\exp(\mathcal{W}_{attn}\odot\mathcal{P}_{m}( \widehat{\bm{e}}_{m}))}{\sum_{n\in\mathcal{M}}\exp(\mathcal{W}_{attn}\odot \mathcal{P}_{n}(\widehat{\bm{e}}_{n}))}\mathcal{P}_{m}(\widehat{\bm{e}}_{m,r})$ (3) $$

where $\mathcal{P}_{m}(m\in\mathcal{M})$ is a projection layer for modality transformation, $\mathcal{W}_{attn}$ is a learnable attention vector shared by each modality, and $\odot$ is the product operator. The joint embedding $\widehat{\bm{e}}_{Joint}$ aggregates information from all modalities and we treat it as another new "modality" $J$ (short for joint).

We further employ Tucker (Balazevic et al., ) score function $\mathcal{S}_{m}(m\in\mathcal{M})$ to measure the triple plausibility from each modality’s perspective, which is denoted as:

$$ $\mathcal{S}_{m}(h,r,t)=\mathcal{W}_{m}\times_{1}\widehat{\bm{h}}_{m,r}\times_{ 2}\bm{r}_{m}\times_{3}\widehat{\bm{t}}_{m,r}$ (4) $$

where $\times_{i}$ represents the tensor product along the i-th mode, $\bm{r}_{m}$ is the learnable embedding of relation $r$ for each modality, $\mathcal{W}_{m}$ is the core tensor learned during training. We train our model with cross-entropy loss for each triple. For a given triple $(h,r,t)$, we treat $t$ as the golden label for tail prediction against the whole entity set $\mathcal{E}$ and $h$ as the golden label for head prediction, which is the negative sampling process mentioned before. The training objective of each modality $m\in\mathcal{M}\cup\{J\}$ can be denoted as:

$$ $\mathcal{L}_{m}=-\sum_{(h,r,t)\in\mathcal{T}}\left(\log\frac{\exp(\mathcal{S}_ {m}(h,r,t))}{\sum_{h^{\prime}\in\mathcal{E}}\exp(\mathcal{S}_{m}(h^{\prime},r, t))}+\log\frac{\exp(\mathcal{S}_{m}(h,r,t))}{\sum_{t^{\prime}\in\mathcal{E}} \exp(\mathcal{S}_{m}(h,r,t^{\prime}))}\right)$ (5) $$

This is the standard KGC model training objective, which MuJoD extends to train a separate scoring function for each modality. The overall training objective of MuJoD can be denoted as:

$$ $\mathcal{L}_{kgc}=\sum_{m\in\mathcal{M}\cup\{J\}}\mathcal{L}_{m}$ (6) $$

Since these objectives from different modalities have consistent prediction targets, we directly combine them to derive the final loss $\mathcal{L}_{kgc}$. In the design of MoMoK, we construct intra-modality experts to learn relation-guided embeddings in ReMoKE, and further collectively combine these inter-modality decisions to make more thoughtful predictions. Each modality serves as a senior expert, making decisions in collaboration with the insights from the intra-modality junior experts (single networks in ReMoKE). This hierarchical expert network architecture enables the progressive delivery of valuable entity modal information.

### 4.3 Expert Information Disentanglement

Additionally, to further allow the model to learn multi-perspective embeddings guided by the relational context, we propose another expert information disentanglement (ExID) module to disentangle the experts’ decisions in each modality based on contrastive log-ratio upper bound (CLUB) (Cheng et al., ), which minimizes the mutual information between the multi-perspective embeddings for each modality using CLUB. The general formulation of CLUB can be denoted as:

$$ $\mathrm{I}_{\mathrm{CLUB}}(\bm{x};\bm{y}):=\mathbb{E}_{p(\bm{x},\bm{y})}[\log p (\bm{y}\mid\bm{x})]-\mathbb{E}_{p(\bm{x})}\mathbb{E}_{p(\bm{y})}[\log p(\bm{y} \mid\bm{x})]$ (7) $$

where $\bm{x};\bm{y}$ are two random variables. $\mathrm{I}_{\mathrm{CLUB}}(\bm{x};\bm{y})$ is an upper bound estimation of their mutual information $\mathrm{I}(\bm{x};\bm{y})$. By optimizing such an objective, we can achieve information disentangling between $\bm{x}$ and $\bm{y}$. Since the true conditional probability distribution $p(\bm{y}\mid\bm{x})$ is difficult to observe, we can use a variational approximation network to approximate this probability distribution. For the above analysis, we provide more detailed formula derivations and proofs in the Appendix A.
In our implementation, for modality $m$ with $K$ MoKEs, we disentangle the multi-perspective embeddings $\mathcal{V}^{e}_{m,i}(1\leq i\leq K)$ of K MoKEs from each other by the following CLUB mutual information estimation:

$$ $\mathcal{L}_{club}=\frac{1}{K^{2}}\sum_{m\in\mathcal{M}}\sum_{e\in\mathcal{B}} \sum_{i=1}^{K}\sum_{j\not=i}^{K}\left(\log\mathcal{Q}_{\theta,m}(\mathcal{V}^{ e}_{m,j}|\mathcal{V}^{e}_{m,i})-\sum_{e^{\prime}\in\mathcal{B}-\{e\}}\log \mathcal{Q}_{\theta,m}(\mathcal{V}^{e^{\prime}}_{m,j}|\mathcal{V}^{e}_{m,i})\right)$ (8) $$

where $\mathcal{B}$ is a batch of entities and $\mathcal{Q}_{\theta,m}(y|x)$ is the variational approximation of ground-truth posterior of $y$ given $x$ parameterized by a neural network $\theta$ for modality $m$. $e^{\prime}$ is another entity sampled from the batch $\mathcal{B}$. With such contrastive loss, we can then make MoKEs minimize the mutual information between decisions. Meanwhile, $\mathcal{Q}_{\theta,m}$ should also be trained to o minimize the KL-divergence between the real conditional probabilities distribution $P(\mathcal{V}^{e}_{m,j}|\mathcal{V}^{e}_{m,i})$ and the variational approximation $\mathcal{Q}_{\theta,m}(\mathcal{V}^{e}_{m,j}|\mathcal{V}^{e}_{m,i})$ by optimize the Kullback-Leibler divergence:

$$ $\mathcal{L}_{exid}=\mathbb{D}_{KL}\left[P(\mathcal{V}^{e}_{m,j}|\mathcal{V}^{e }_{m,i})||\mathcal{Q}_{\theta,m}(\mathcal{V}^{e}_{m,j}|\mathcal{V}^{e}_{m,i})\right]$ (9) $$

which will be alternatively optimized with the main MMKGC model during training. Here, the real conditional distribution is usually assumed as a Gaussian distribution (Cheng et al., ).

### 4.4 Training and Inference

Combining all the designs above, the final objective for our MMKGC model can be represented as:

$$ $\mathcal{L}=\mathcal{L}_{kgc}+\lambda\mathcal{L}_{club}$ (10) $$

We collectively train the embeddings with prediction losses $\mathcal{L}_{m}$ from each modality in a multi-task manner. The disentangle loss $\mathcal{L}_{club}$ is regulated by a weight $\lambda$. Besides, during each round of training, $\mathcal{Q}_{\theta,m}$ is also optimized with the loss $\mathcal{L}_{exid}$, separated from the MMKGC model. During the inference stage, we calculate the joint score for each triple as $\mathcal{S}(h,r,t)=\sum_{m\in\mathcal{M}\cup\{J\}}\mathcal{S}_{m}(h,r,t)$
which considers the contribution from each modality and provides a full-view prediction. This score function $\mathcal{S}(h,r,t)$ will be the final measurement of the triple plausibility and applied for candidate triple ranking and evaluation.

**Table 1: Statistical information of the four MMKGs in our experiments. The image and text modality features are provided by the original datasets and kept the same for all baselines.**
| Dataset | #Entity | #Relation | #Train | #Valid | #Test | Image | Text |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Num | Dim | Num | Dim |  |  |  |  |  |  |
| MKG-W (Xu et al., ) | 15000 | 169 | 34196 | 4276 | 4274 | 14463 | 383 | 14123 | 384 |
| MKG-Y (Xu et al., ) | 15000 | 28 | 21310 | 2665 | 2663 | 14244 | 383 | 12305 | 384 |
| DB15K (Liu et al., ) | 12842 | 279 | 79222 | 9902 | 9904 | 12818 | 4096 | 9078 | 768 |
| KVC16K (Zhang et al., ) | 16015 | 4 | 180190 | 22523 | 22525 | 14822 | 768 | 14822 | 768 |

## 5 Experiments and Evaluation

In this section, we will introduce the basic experiment settings of our work and demonstrate our evaluation results with extensive analysis.
The following five research questions (RQ) are the key questions that we explore in the experiments.

- RQ1.
Can MoMoK outperform the existing baselines and achieve state-of-the-art performance?
- RQ2.
Can MoMoK maintain robust performance tasks when the modality information is noisy?
- RQ3.
How much do each module in the MoMoK contribute to the final performance?
- RQ4.
How does the training efficiency of our model compare to existing methods?
- RQ5.
Are there intuitive cases to straightly demonstrate the effectiveness of MoMoK?

### 5.1 Datasets

We conduct our experiments on four public MMKG benchmarks: MKG-W (Xu et al., ), MKG-Y (Xu et al., ), DB15K (Liu et al., ), and KVC16K (Zhang et al., ). MKG-W and MKG-Y are the subsets of Wikidata (Vrandecic & Krötzsch, ), YAGO (Suchanek et al., ), and DBPedia (Lehmann et al., ) respectively. KVC16K is modified from KuaiPedia (Pan et al., ), a micro-video encyclopedia. They are all real-world MMKGs with image and text modalities. The detailed information on the datasets can be found in Table 1.

### 5.2 Experimental Settings

#### Baseline Methods

To make comprehensive comparisons, we chose 20 recent SOTA MMKGC methods as the baselines for the experiments. The first category is uni-modal KGC methods including TransE (Bordes et al., ), DistMult (Yang et al., ), ComplEx (Trouillon et al., ), RotatE (Sun et al., ), PairRE (Chao et al., ), and TuckER (Balazevic et al., ). The second category is multi-modal KGC methods considering multi-modal information of entities to enhance the KGC models, including IKRL (Xie et al., ), TBKGC (Sergieh et al., ), TransAE (Wang et al., ), RSME (Wang et al., ), MMKRL (Lu et al., ), VBKGC (Zhang & Zhang, ), OTKGE (Cao et al., ), MoSE (Zhao et al., ), MMRNS (Xu et al., ), MANS (Zhang et al., ), IMF (Li et al., ), QEB (Wang et al., ), VISTA (Lee et al., ), and AdaMF (Zhang et al., ). Among these methods, MoSE and IMF are two methods using ensemble learning technologies, which have similarities to our design and are worth making comparisons. The third category is negative sampling methods including MANS (Zhang et al., ) and MMRNS (Xu et al., ). Please refer to Appendix B.1 for more detailed information.

**Table 2: The main MMKGC results on four datasets. The best results are bold and the second best results are underlined. We report the improvement of MoMoK compared to the optimal baseline. Methods with special mark * are ensemble-based methods considering integrating different modalities.**
| Model | MKG-W | MKG-Y | DB15K | KVC16K |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MRR | Hit@1 | MRR | Hit@1 | MRR | Hit@1 | Hit@3 | Hit@10 | MRR | Hit@1 | Hit@3 | Hit@10 |  |
| Uni-modal KGC Methods |  |  |  |  |  |  |  |  |  |  |  |  |
| TransE (Bordes et al., ) | 29.19 | 21.06 | 30.73 | 23.45 | 24.86 | 12.78 | 31.48 | 47.07 | 8.54 | 0.64 | 10.97 | 23.42 |
| DistMult (Yang et al., ) | 20.99 | 15.93 | 25.04 | 19.33 | 23.03 | 14.78 | 26.28 | 39.59 | 6.37 | 3.03 | 6.11 | 12.61 |
| ComplEx (Trouillon et al., ) | 24.93 | 19.09 | 28.71 | 22.26 | 27.48 | 18.37 | 31.57 | 45.37 | 12.85 | 7.48 | 13.79 | 23.18 |
| RotatE (Sun et al., ) | 33.67 | 26.80 | 34.95 | 29.10 | 29.28 | 17.87 | 36.12 | 49.66 | 14.33 | 8.25 | 15.37 | 26.17 |
| PairRE (Chao et al., ) | 34.40 | 28.24 | 32.01 | 25.53 | 31.13 | 21.62 | 35.91 | 49.30 | - | - | - | - |
| TuckER (Balazevic et al., ) | 29.59 | 23.93 | 37.05 | 34.59 | 33.86 | 25.34 | 37.91 | 50.38 | 15.90 | 9.79 | 17.24 | 27.58 |
| Multi-modal KGC Methods |  |  |  |  |  |  |  |  |  |  |  |  |
| IKRL (Xie et al., ) | 32.36 | 26.11 | 33.22 | 30.37 | 26.82 | 14.09 | 34.93 | 49.09 | 11.11 | 5.42 | 11.46 | 22.39 |
| TBKGC (Sergieh et al., ) | 31.48 | 25.31 | 33.99 | 30.47 | 28.40 | 15.61 | 37.03 | 49.86 | 5.39 | 0.35 | 5.04 | 15.52 |
| TransAE (Wang et al., ) | 30.00 | 21.23 | 28.10 | 25.31 | 28.09 | 21.25 | 31.17 | 41.17 | 10.81 | 5.31 | 11.34 | 21.89 |
| MMKRL (Lu et al., ) | 30.10 | 22.16 | 36.81 | 31.66 | 26.81 | 13.85 | 35.07 | 49.39 | 8.78 | 3.89 | 8.99 | 18.34 |
| RSME (Wang et al., ) | 29.23 | 23.36 | 34.44 | 31.78 | 29.76 | 24.15 | 32.12 | 40.29 | 12.31 | 7.14 | 13.21 | 22.05 |
| VBKGC (Zhang & Zhang, ) | 30.61 | 24.91 | 37.04 | 33.76 | 30.61 | 19.75 | 37.18 | 49.44 | 14.66 | 8.28 | 15.81 | 27.04 |
| OTKGE (Cao et al., ) | 34.36 | 28.85 | 35.51 | 31.97 | 23.86 | 18.45 | 25.89 | 34.23 | 8.77 | 5.01 | 9.31 | 15.55 |
| MoSE* (Zhao et al., ) | 33.34 | 27.78 | 36.28 | 33.64 | 28.38 | 21.56 | 30.91 | 41.67 | 8.81 | 4.75 | 9.46 | 16.40 |
| IMF* (Li et al., ) | 34.50 | 28.77 | 35.79 | 32.95 | 32.25 | 24.20 | 36.00 | 48.19 | 12.01 | 7.42 | 12.82 | 21.01 |
| QEB (Wang et al., ) | 32.38 | 25.47 | 34.37 | 29.49 | 28.18 | 14.82 | 36.67 | 51.55 | 12.06 | 5.57 | 13.03 | 25.01 |
| VISTA (Lee et al., ) | 32.91 | 26.12 | 30.45 | 24.87 | 30.42 | 22.49 | 33.56 | 45.94 | 11.89 | 6.97 | 12.66 | 21.27 |
| AdaMF (Zhang et al., ) | 34.27 | 27.21 | 38.06 | 33.49 | 32.51 | 21.31 | 39.67 | 51.68 | 15.26 | 8.56 | 16.71 | 28.29 |
| Negative Sampling Methods |  |  |  |  |  |  |  |  |  |  |  |  |
| MANS (Zhang et al., ) | 30.88 | 24.89 | 29.03 | 25.25 | 28.82 | 16.87 | 36.58 | 49.26 | 10.42 | 5.21 | 11.01 | 20.45 |
| MMRNS (Xu et al., ) | 35.03 | 28.59 | 35.93 | 30.53 | 32.68 | 23.01 | 37.86 | 51.01 | 13.31 | 7.51 | 14.19 | 24.68 |
| MoMoK | 35.89 | 30.38 | 37.91 | 35.09 | 39.57 | 32.38 | 43.45 | 54.14 | 16.87 | 10.53 | 18.26 | 29.20 |
| Improvements | +2.5% | +4.2% | - | +3.9% | +21.1% | +33.8% | +9.5% | +4.8% | +10.6% | +23.0% | +9.3% | +3.21% |

#### Task and Evaluation Protocols

We evaluate the MMKGC models with the link prediction task (Bordes et al., ), the most popular KGC task. We use rank-based metrics like mean reciprocal rank (MRR) (Sun et al., ), and Hit@K (K=1, 3, 10)(Bordes et al., ) to quantitatively evaluate the link prediction performance, considering both head prediction $(h,r,?)$ and tail prediction $(?r,t)$. Filter setting (Bordes et al., ) is used to eliminate the effect of triples that have already appeared in the training data. Please refer to Appendix B.2 for more detailed information.

#### Implemention Details

In our experiments, we implement our method with PyTorch and conduct each experiment on a Linux server with the Ubuntu 20.04.1 operating system and a single NVIDIA A800 GPU. The variational approximation network $\theta$ and the projection layers are all implemented by two-layer MLPs with ReLU as activation (Glorot et al., ). During training, we set the batch size to 1024. The embedding dimension $d$ is tuned from $\{200,250,300\}$. We optimize the model with Adam (Kingma & Ba, ) and the learning rate is tuned from $\{1e^{-3},5e^{-4},1e^{-4}\}$. The loss weight $\lambda$ is tuned in $\{1e^{-3},1e^{-4},1e^{-5}\}$. Each single experiment takes 1-3 hours to accomplish across different datasets. For baselines, we reproduce the results following the settings described in the original papers and their open-source official code. Some of the baseline results refer to MMRNS (Xu et al., ). Please refer to Appendix B.3 for more detailed information about the hyper-parameters.

### 5.3 Main Results (RQ1)

The main MMKGC results are detailed in Table 2. Comparison with the recent 19 baselines reveals that MoMoK makes significant progress in almost all the metrics and achieves new state-of-the-art results. When contrasted with existing ensemble-based approaches such as MoSE (Zhao et al., ) and IMF (Li et al., ), MoMoK excels by fully exploiting the potential of the relational context. These methods often merely assign weights to models across different modalities, overlooking the impact of intricate factors like relational context. In contrast, MoMoK thoroughly incorporates these considerations.

Furthermore, it is evident that MoMoK achieves most pronounced improvements in Hit@1 across different metrics. For instance, MoMoK obtained 33.8% / 23.0% relative improvement of Hit@1 on DB15K and KVC16K respectively. This underscores that, compared with baseline models, our method is more effective at ranking correct answers first. It demonstrates the significant contribution of our multi-modal information utilization and relational context to the refined, accurate reasoning capabilities of the MMKGC model.

Figure: Figure 3: MMKGC results (MRR and Hit@10) of DB15K dataset under three different scenario: modality noisy, modality missing, and link sparse. We compare our method MoMoK with three recent MMKGC baselines AdaMF, TBKGC, and QBE.
Refer to caption: x3.png

### 5.4 MMKGC Experiments in Complex Environments (RQ2)

To assess the robustness of our method in complex scenarios, we conducted MMKGC experiments in three different scenarios: modality noisy scenario, modality missing scenario, and link sparse scenario. For the modality noisy scenario, we set a noise ratio for the MMKG dataset, according to which a portion of the solid modal information is added with Gaussian noise before performing the MMKGC experiments. For the modality missing scenario, we randomly remove some of the modality information for a certain proportion of entities. For a link sparse scenario, we randomly remove some of the training triples and create a data-sparse environment.

As presented in Figure 3, the experimental results indicate that MoMoK continues to outperform the baseline method despite these conditions. It is evident that the complex environments significantly affect the MMKGC prediction at coarse grains, as indicated by the more pronounced volatility of Hit@10 compared to MRR. The variation in Hit@10 results reveals that baseline methods like TBKGC (Sergieh et al., ) and AdaMF (Zhang et al., ) undergo a noticeable performance degradation with increasing noise and decreasing training data, while our method’s performance remains relatively steady. From the above implementation results, our approach achieves better results compared to baseline in a variety of complex scenarios.

**Table 3: The ablation study results on MKG-W and DB15K datasets. We explored the impact of the design of each modality already each important component on the final result.**
| Setting | MKG-W | DB15K |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MRR | Hit@1 | MRR | Hit@1 | Hit@3 | Hit@10 |  |  |
| Full Model | 35.89 | 30.38 | 39.57 | 32.38 | 43.45 | 54.14 |  |
| Modality<br>Contribution | (1.1). Structure Modality | 32.82 | 27.73 | 36.45 | 29.36 | 39.99 | 49.86 |
| (1.2). Image Modality | 32.75 | 27.78 | 36.84 | 29.80 | 40.10 | 50.42 |  |
| (1.3). Text Modality | 32.62 | 27.66 | 37.04 | 29.93 | 40.49 | 50.39 |  |
| (1.4). Joint Modality | 34.76 | 29.33 | 36.87 | 29.90 | 42.44 | 53.93 |  |
| Model<br>Design | (2.1). w/o relational $\epsilon_{r}$ | 35.50 | 29.98 | 39.40 | 31.47 | 43.19 | 52.88 |
| (2.2). w/o noise $\delta_{m}$ | 35.31 | 29.69 | 39.43 | 31.54 | 43.32 | 53.75 |  |
| (2.3). w/o adaptive fusion | 35.34 | 30.04 | 39.01 | 30.74 | 43.29 | 53.85 |  |
| (2.4). w/o joint training | 32.73 | 27.09 | 37.62 | 29.72 | 41.64 | 52.73 |  |
| (2.5). w/o ExID | 34.99 | 29.49 | 38.42 | 30.63 | 42.42 | 53.24 |  |

Figure: Figure 4: Overview of our proposed framework MoMoK. MoMoK consists of a collaborative pre-training stage and a prefix prompt tuning stage, which first pre-trains on the large-scale multi-domain item KGs and fine-tuned on the item-aware downstream tasks like recommendation and text understandings with a lightweight prefix prompt token.
Refer to caption: x4.png

### 5.5 Ablation Study (RQ3)

To confirm the soundness of our design, we conduct further ablation studies to investigate the contribution of each module in MoMoK. Our ablation experiments are divided into two main parts. The first part aims to analyze the information from each modality and validate whether they positively contribute to the performance. The second part is dedicated to examining our designs in MoMoK (ReMoKE, MuJoD, ExID) and verifying whether their design has rationality by removing the corresponding modules. The experimental results are presented in Table 3.

From the first group of experimental results we can observe that each modality’s information contributes to the final result, and during training we set up a separate model for each modality, with their respective performance on two datasets being lower than the full model result.

Moreover, the results from the second group reveal that some of our key designs in the three modules significantly contribute to the final performance. Experiments (2.1) and (2.2) confirm the effectiveness of relational context and tunable noise in the ReMoKE module. Experiments (2.3) and (2.4) focus on the MuJoD module and the results proved the effectiveness of the adaptive fusion (Equation 3) and joint training (Equation 6). Experiment (2.5) further examines the impact of the CLUB loss on information disentanglement. Collectively, these findings indicate that joint training has the most profound effect on the final performance, as it trains a separate MMKGC model for each modality, resulting in decision fusion.

We also investigate the effect of some crucial hyperparameters, such as the number of experts $K$ in the ReMoKE module, and the weights of the ExID loss $\lambda$, as depicted in Figure 4. It can be observed that the impact of the number of experts K on the final results generally follows a pattern of initial increase followed by a decrease, mainly affecting fine-grained metrics such as Hit@1 and MRR. Having either too many or too few experts is detrimental to the model’s learning performance. The impact of weight $\lambda$ is similar. The model achieves the best results at $K=3$ and $\lambda=0.0001$.

**Table 4: Time and GPU memory cost for different MMKGC methods.**
| Method | Time | GPU Memory |
| --- | --- | --- |
| MoMoK | 9.8s | 5900MB |
| MMKRL | 7.5s | 4504MB |
| OTKGE | 70.1s | 2540MB |
| MMRNS | 25.5s | 25582MB |

### 5.6 Efficiency Analysis (RQ4)

Even though our design introduces many new modules that have not appeared in existing methods, there is not much loss of efficiency in terms of time and space complexity.
The introduction of new computational and parametric quantities occurs in two main sections, the MuJoD module and the ExID module. Our approach remains linear in the growth of complexity, and there is no such level of latency as an exponential explosion. Here, we present the time efficiency and GPU memory usage in Table 4. Combining the SOTA MMKGC performance of MoMoK in Table 2 and the training efficiency/GPU consumption results presented in this table we can conclude that our approach achieves high efficiency and less GPU memory consumption while maintaining the SOTA effect. Therefore, the time and space complexity of the present method is within reasonable limits and still performs well.

**Table 5: A case study of MoMoK. We list some of the relations that are predicted best by each modality score $\mathcal{S}_{m}$ to verify the contribution of each modality to the final result.**
| Modality | Outperforming Predictions |
| --- | --- |
| Structure | RouteJunction, PartOf, Nearest City, Publisher, PrimeMinister, ComputingPlatform, LargestCity |
| Image | SisterStation, League, Parent, HubAirport, Company, Owner, Capital |
| Text | SisterStation, Publisher, Head, FederalState, Parent, ComputingPlatform, CountrySeat |
| Joint | GoverningBody, PartOf, Creator, Company, ComputingPlatform, RegionServed |
| Full Model | SisterStation, RouteJunction, GoverningBody, Publisher, PartOf, FederalState, ComputingPlatform, Parent |

Figure: Figure 5: Attention weights visualization results. We select some relations and present the weights of each modality contributing to the joint representation $\widehat{e}_{Joint}$. We further present the weights $G_{i}$ for $K(K=3)$ ReMoKEs in the modality outputs $\widehat{\bm{e}}_{m}$. Abbreviations for modalities: Structure (STR), Image (IMG), Text (TXT). M.k in the legend denotes the k-th expert of modality M.
Refer to caption: x5.png

### 5.7 Case Study (RQ5)

To provide a more intuitive justification and interpretability for our approach, we conduct the case study from both macroscopic and microscopic viewpoints. We set a separate score for each modality in the MuJoD module and finally integrate them for joint decision-making. Therefore, to visualize the contribution of each modality to the final result, we list in Table 5 several relations where each modality score achieves the best results.

Notably, the relation types that each modality score $\mathcal{S}_{m}$ best predicts are diverse. These relations that perform best in the overall prediction can be found in the prediction results of the different modalities. This implies that MoMoK effectively merges predictions from various modalities for joint consideration, thereby outperforming the results achieved by individual modalities.

Simultaneously, we delve into the micro level by analyzing the adaptive weights in MoMoK. Our design incorporates the expert decisions via a series of adaptive weights in the ReMoKE, while the MuJoD module also employs adaptive weights to derive the joint modality embedding from the outputs of the modalities. We select a handful of relations to investigate the weights from each modality and each ReMoKE within the joint modality embedding of the entities in the respective relational contexts.

As shown in Figure 5, the joint embedding of entities in varied relational contexts assigns diverse significance to each modality’s information. For example, PartOf attaches more weight to textual modality, while Parents relies more on image modality. Furthermore, the majority of contributions within each modality come from the same expert, and the contributions from different modalities in distinct relational contexts vary greatly. This indicates that we successfully delegate different ReMoKEs intra-modality to handle different relational contexts, aligning with our original intent of proposing the MoE architecture to address the MMKGC task.

## 6 Conclusion and Future Work

In this paper, we propose a new MMKGC framework called MoMoK to learn modality features in diverse perspectives from the raw modality information of entities with relational guidance and integrate the multi-modal information through modality knowledge experts. We further decouple the expert networks and enhance the model’s expressive capability through the comparative estimation of mutual information. Experimental results show that our design can achieve new SOTA results on multiple public benchmarks with both robustness, reasonability, and interpretability. Looking ahead, we can further design a more rational MoE architecture that not only accomplishes the tasks of the MMKGC but also finds ways to incorporate the MMKG and the large language models to realize a sparse large language model with multi-modal knowledge perception.

## Appendix A Model Design and Analysis

In this section, we present a detailed theoretical analysis of the CLUB loss we used in Section 4.3.

#### Theorem 1

: For two random variables $\bm{x},\bm{y}$, the upper bound of their mutual information $\mathrm{I}(\bm{x};\bm{y})$ is:

$$ $\mathrm{I}_{\mathrm{CLUB}}(\bm{x};\bm{y}):=\mathbb{E}_{p(\bm{x},\bm{y})}[\log p (\bm{y}\mid\bm{x})]-\mathbb{E}_{p(\bm{x})}\mathbb{E}_{p(\bm{y})}[\log p(\bm{y} \mid\bm{x})]$ (11) $$

#### Proof 1

This theorem can be proven with the following derivation. We calculate the difference between $\mathrm{I}_{\mathrm{CLUB}}(\bm{x};\bm{y})$ and $\mathrm{I}(\bm{x};\bm{y})$ by:

$$ $\displaystyle\Delta:$ $\displaystyle=\mathrm{I}_{\mathrm{CLUB}}(\bm{x};\bm{y})-\mathrm{I}(\bm{x};\bm{ y})$ (12) $\displaystyle=\mathbb{E}_{p(\bm{x},\bm{y})}[\log p(\bm{y}\mid\bm{x})]-\mathbb{ E}_{p(\bm{x})}\mathbb{E}_{p(\bm{y})}[\log p(\bm{y}\mid\bm{x})]-\mathbb{E}_{p( \bm{x},\bm{y})}[\log p(\bm{y}\mid\bm{x})-\log p(\bm{y})]$ $\displaystyle=\mathbb{E}_{p(\bm{x},\bm{y})}[\log p(\bm{y})]-\mathbb{E}_{p(\bm{ x})}\mathbb{E}_{p(\bm{y})}[\log p(\bm{y}\mid\bm{x})]=\mathbb{E}_{p(\bm{y})}[ \log p(\bm{y})-\mathbb{E}_{p(\bm{x})}[\log p(\bm{y}\mid\bm{x})]]$ $$

Besides, according to Jensen inequality, we have:

$$ $\displaystyle\log p(\bm{y})=$ $\displaystyle\log(\mathbb{E}_{p(\bm{x})}[p(\bm{y}\mid\bm{x})])\geq\mathbb{E}_{ p(\bm{x})}[\log p(\bm{y}\mid\bm{x})]$ (13) $\displaystyle\Rightarrow\Delta:$ $\displaystyle=\mathrm{I}_{\mathrm{CLUB}}(\bm{x};\bm{y})-\mathrm{I}(\bm{x};\bm{ y})\geq 0$ $$

Therefore, $\mathrm{I}_{\mathrm{CLUB}}(\bm{x};\bm{y})$ is an upper bound of $\mathrm{I}(\bm{x};\bm{y})$. Besides, the real conditional distribution $p(\bm{y}\mid\bm{x})$ is hard to measure. Therefore, a variational distribution $q_{\theta}(\bm{y}\mid\bm{x})$ parameterized by a neural network $\theta$ is usually employed to approximate the real distribution. Therefore, the variational CLUB objective can be denoted as:

$$ $\mathrm{I}^{\prime}_{\mathrm{CLUB}}(\bm{x};\bm{y}):=\mathbb{E}_{q_{\theta}(\bm {x},\bm{y})}[\log q_{\theta}(\bm{y}\mid\bm{x})]-\mathbb{E}_{q_{\theta}(\bm{x}) }\mathbb{E}_{q_{\theta}(\bm{y})}[\log q_{\theta}(\bm{y}\mid\bm{x})]$ (14) $$

It can be proved that with a good variational network $q_{\theta}$, $\mathrm{I}^{\prime}$ can still hold an upper bound of the mutual information. Here we denote $q_{\theta}(\bm{x},\bm{y})=q_{\theta}(\bm{y}\mid\bm{x})p(\bm{x})$ and introduce such a new theorem:

#### Theorem 2

If we have

$$ $KL(p(\bm{x},\bm{y})\|q_{\theta}(\bm{x},\bm{y}))\leq KL(p(\bm{x})p(\bm{y})\|q_{ \theta}(\bm{x},\bm{y}))$ (15) $$

then we have $\mathrm{I}(\bm{x};\bm{y})\leq\mathrm{I}^{\prime}_{\mathrm{CLUB}}(\bm{x};\bm{y})$. KL represents the Kullback-Leibler divergence between two probability distributions.

#### Proof 2

We can have the following derivation:

$$ $\displaystyle\begin{aligned} \tilde{\Delta}&:=\mathrm{I}^{\prime}_{\mathrm{ CLUB}}(\bm{x};\bm{y})-\mathrm{I}(\bm{x};\bm{y})\\ &=\mathbb{E}_{p(\bm{x},\bm{y})}[\log q_{\theta}(\bm{y}\mid\bm{x})]-\mathbb{E}_ {p(\bm{x})}\mathbb{E}_{p(\bm{y})}[\log q_{\theta}(\bm{y}\mid\bm{x})]-\mathbb{E }_{p(\bm{x},\bm{y})}[\log p(\bm{y}\mid\bm{x})-\log p(\bm{y})]\\ &=[\mathbb{E}_{p(\bm{y})}[\log p(\bm{y})]-\mathbb{E}_{p(\bm{x})p(\bm{y})}[\log q _{\theta}(\bm{y}\mid\bm{x})]\\ &-[\mathbb{E}_{p(\bm{x},\bm{y})}[\log p(\bm{y}\mid\bm{x})]-\mathbb{E}_{p(\bm{x },\bm{y})}[\log q_{\theta}(\bm{y}\mid\bm{x})]].\\ &=\mathbb{E}_{p(\bm{x})p(\bm{y})}[\log\frac{p(\bm{y})}{q_{\theta}(\bm{y}\mid \bm{x})}]-\mathbb{E}_{p(\bm{x},\bm{y})}[\log\frac{p(\bm{y}\mid\bm{x})}{q_{ \theta}(\bm{y}\mid\bm{x})}]\\ &=\mathbb{E}_{p(\bm{x})p(\bm{y})}[\log\frac{p(\bm{x})p(\bm{y})}{q_{\theta}(\bm {y}\mid\bm{x})p(\bm{x})}]-\mathbb{E}_{p(\bm{x},\bm{y})}[\log\frac{p(\bm{y}\mid \bm{x})p(\bm{x})}{q_{\theta}(\bm{y}\mid\bm{x})p(\bm{x})}]\\ &=\operatorname{KL}(p(\bm{x})p(\bm{y})\|q_{\theta}(\bm{x},\bm{y}))- \operatorname{KL}(p(\bm{x},\bm{y})\|q_{\theta}(\bm{x},\bm{y}))\end{aligned}$ (16) $$

Therefore, $\mathrm{I}^{\prime}_{\mathrm{CLUB}}(\bm{x};\bm{y})$ is an upper bound of $\mathrm{I}(\bm{x};\bm{y})$ if and only if $KL(p(\bm{x},\bm{y})\|q_{\theta}(\bm{x},\bm{y}))\leq KL(p(\bm{x})p(\bm{y})\|q_{
\theta}(\bm{x},\bm{y}))$. The equality holds when $\bm{x},\bm{y}$ are independent.

This theorem indicates that $\mathrm{I}^{\prime}_{\mathrm{CLUB}}(\bm{x};\bm{y})$ can still be an upper bound for $\mathrm{I}(\bm{x};\bm{y})$ if the variational distribution $q_{\theta}(\bm{x},\bm{y})$ is closer to $p(\bm{x},\bm{y})$ than to $p(\bm{x})p(\bm{y})$. Hence, we can minimizing $KL(p(\bm{x},\bm{y})||q_{\theta}(\bm{x},\bm{y}))$ to achieve this goal.

In our specific implementations, we employ several neural networks $\mathcal{Q}_{\theta,m}$ to estimate the variational distribution. We further design two loss ($\mathcal{L}_{club}$ in Equation 8 and $\mathcal{L}_{exid}$ in Equation 9) to implement the CLUB module. $\mathcal{L}_{club}$ is designed to estimate the mutual information among different MoKEs and $\mathcal{L}_{exid}$ is designed to optimize the neural networks $\mathcal{Q}_{\theta,m}$. These two losses correspond to Theorem 1 and Theorem 2 derived earlier.

## Appendix B Experiment Details

### B.1 Baselines

The KGC baselines we compared in our experiments can be divided into three categories:

(1). Uni-modal KGC methods: TransE (Bordes et al., ), DistMult (Yang et al., ), ComplEx (Trouillon et al., ), RotatE (Sun et al., ), PairRE (Chao et al., ), and TuckER (Balazevic et al., ). These methods only consider the triple structural information in their designs and learn structural embeddings for KGC.

(2). Multi-modal KGC methods: IKRL (Xie et al., ), TBKGC (Sergieh et al., ), TransAE (Wang et al., ), RSME (Wang et al., ), MMKRL (Lu et al., ), VBKGC (Zhang & Zhang, ), OTKGE (Cao et al., ), MoSE (Zhao et al., ), MMRNS (Xu et al., ), MANS (Zhang et al., ), IMF (Li et al., ), QEB (Wang et al., ), VISTA (Lee et al., ), and AdaMF (Zhang et al., ). These methods employ multi-modal information such as texts and images to enhance the entity representation learning, achieving better KGC performance.

(3). Negative sampling methods: MANS (Zhang et al., ) and MMRNS (Xu et al., ). These methods attempt to modify the traditional negative sampling methods with the multi-modal information from entities to make fine-grained contrast during training.

In the experiments, we conduct a comprehensive comparison with the mentioned 20 baselines to demonstrate that MoMoK can learn better entity representations to enhance the MMKGC process. Some methods (Yao et al., ) fine-tuning the pre-trained models are orthogonal to our design philosophy and paradigm so we do not compare with them.

### B.2 Evaluation Protocols

We conduct a link prediction task on the datasets, the mainstream MMKGC task. Following existing works, we use rank-based metrics like mean reciprocal rank (MRR) and Hit@K (K=1, 3, 10) to evaluate the results. Besides, we employ the filter setting in the prediction results to remove the candidate triples that already appeared in the training data for fair comparisons. The final results are the average of both head prediction and tail prediction. MRR and Hit@K can be calculated as:

$$ $\mathbf{MRR}=\frac{1}{|\mathcal{T}_{test}|}\sum_{i=1}^{|\mathcal{T}_{test}|}( \frac{1}{r_{h,i}}+\frac{1}{r_{t,i}})$ (17) $$

$$ $\mathbf{Hit@K}=\frac{1}{|\mathcal{T}_{test}|}\sum_{i=1}^{|\mathcal{T}_{test}|} (\mathbf{1}(r_{h,i}\leq K)+\mathbf{1}(r_{t,i}\leq K))$ (18) $$

where $r_{h,i}$ and $r_{t,i}$ are the results of head prediction and tail prediction respectively, $\mathcal{T}_{test}$ is the test triple set.

### B.3 Implementation

We list the hyper-parameters we set in the experiments for reproducing our results in Table 6.

**Table 6: The hyper-parameters for reproducing our experiments.**
| Hyper-parameters | MKG-W | MKG-Y | DB15K | KVC16K |
| --- | --- | --- | --- | --- |
| learning rate | 1e-3 | 5e-4 | 1e-3 | 1e-3 |
| embedding dimension | 200 | 200 | 300 | 300 |
| loss weight $\lambda$ | 1e-4 | 1e-4 | 1e-4 | 1e-4 |
| batch size | 1024 | 1024 | 1024 | 1024 |
| expert number | 3 | 3 | 3 | 3 |
| training epochs | 2000 | 2000 | 2000 | 2000 |

## Appendix C Limitations of Our Work

Our main work is the design and implementation of a novel MMKGC framework. Of course, there are some limitations to our work. The main points are as follows:

- •
Limitations of task scenarios. Our research focuses on a specific research field called MMKGC and our method MoMoK is designed specifically for this task. We do not generalize this framework to more multi-modal tasks.
- •
Integration with LLM trends. Our approach uses a classical embedding-based approach in studying the MMKGC problem and does not combine the MMKG with the latest LLM trends in a synergistic way.
- •
Limitations of the experiment. Due to the lack of standard datasets for super large-scale experiments at MMKG, our experiments were conducted mainly on medium-sized datasets.

## Appendix D Broader Impacts of Our Work

Our work focuses on reasoning about multi-modal knowledge, which can help us discover new possible associations in large-scale semantic networks and encyclopedic knowledge and expand existing encyclopedic knowledge bases such as Wikidata, etc. The positive social impact of our work is to help build and expand the Internet knowledge-sharing community, and to make more accumulation and deposition of linked data. We do not believe that our research will have a negative social impact, and we will also take active steps to avoid misuse of our methods.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: please see above.
soundness: 2
strengths: 1. Experiments in complex environments demonstrate MoMoK's resilience to noise, data sparsity, and missing modalities.  
2. This work introduces a novel expert information disentanglement (ExID) module to separate expert decisions for each modality.  
3. The rationale for designing the relation-aware MKGC method is evident.
summary: This paper tackles MKGE problems by proposing a Mixture of Modality Knowledge Experts framework. The authors develop relation-guided modality knowledge experts to obtain relation-aware modality embeddings while reducing mutual information among the experts. Empirical studies validate the model's effectiveness.
weaknesses: 1. Metrics Reporting: Table 2 displays only the Hit@1 and MRR metrics for MKG-W and MKG-Y, showing modest performance improvements for MoMoK. Please include additional metrics for MoMoK, AdaMF, and MMRNS to better assess the reported gains.

2. MoMoK on Classic MMKGC Datasets: Why wasn't MoMoK evaluated on classic MMKGC datasets like WN9-IMG, FB-IMG, WN18-IMG, and FB15K-237-IMG?

3. Modal Data and Encoder Details: The notation $\mathcal{X}_m(e)$ indicates information from multiple modalities, yet four datasets are used. Please clarify the specific modalities for each dataset, how multiple images are handled, and whether VGG and BERT are consistently used as encoders.

4. Notation Standardization: Some formulas, such as $\mathcal{E}_h$ in formula (1), could benefit from standardized notation to improve clarity and consistency.

5. Relation-Aware Temperature ($\mathcal{E}_r$): The relation-aware temperature $\mathcal{E}_r$ is important for GFN computation, but its definition is absent. Please provide its formal definition and its impact on model performance.

6. Score Function Comparison: The proposed method employs Tucker decomposition as its score function, while the baseline methods utilize various other functions.

7. Ablation Study: Ablation experiments indicate that components like adaptive fusion, noise $\delta_{r}$, and relational $\epsilon_{m}$ may have minimal individual impacts. Further analysis or emphasis on their significance in different experimental contexts is needed.

8. Ablation Experiments for MKG-Y and KVC16K: Can MKG-Y and KVC16K datasets be included in the ablation experiments? This would enhance the assessment of the method's performance across diverse data and conditions.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The parameter \lambda seems small enough. Can you explain why club loss can improve the model by about one point in almost every metric under a weight of 1e-4 (Table 3)?
2. The method seems to be insufficiently integrated with the multimodal large model. This paper uses BERT and VGG to extract the original features of the model. Have you tried to use a multimodal large model to represent them uniformly? I would like to know the performance comparison between the embedding representation of the multimodal large model and MoMoK.
soundness: 3
strengths: 1. This paper considers the modal information hidden in different relational contexts and designs a relation-guided modal knowledge expert network to adaptively aggregate multi-view embeddings.
2. On four public MMKG benchmarks, MoMoK significantly outperforms recent baseline methods and demonstrates its superior performance in complex scenarios.
3. Through detailed ablation analysis, this paper verifies the effectiveness of the design and provides intuitive case studies to enhance the interpretability of the model.
summary: This paper proposes a novel framework, Mixture of Modality Knowledge Experts (MoMoK), which aims to improve the quality of multimodal entity representation learning through relation guidance and expert information decoupling in multimodal knowledge graphs (MMKGs). MoMoK achieves better triple prediction by designing a relation-guided modal knowledge expert network, adaptively aggregating multi-view embeddings, and adopting a multimodal joint decision mechanism. Experimental results show that MoMoK performs well on four public MMKG benchmarks and significantly outperforms existing methods.
weaknesses: 1. Table 2 lacks a comparison of the number of model parameters. Although the authors have discussed the training memory consumption and time efficiency of some methods in Section 5.6, it is beneficial to supplement these values for all baseline models in Table 2 for fair comparison.
2. The description in Table 4 does not seem to be that comprehensive. Does the GPU memory here refer to the memory during inference or the memory during training? Does the time in seconds represent the time it takes to infer one sample?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - Please make a response to the comments in weaknesses.
- What exactly are the advantages of the traditional embedding-based approach over the LLM-based approach for knowledge graph completion?
soundness: 3
strengths: - Multi-modal knowledge graph representation learning is a hot topic in the KG community.
- The three modules in the design of the method proposed by the authors are self-consistent and the authors provide a more detailed theoretical proof.
- The experimental part of the workload is sufficient, and the authors have done more experiments to prove the effectiveness, robustness, and efficiency of the method. Detailed case study and visualization results are also provided.
summary: This paper proposes a multi-modal mixture-of-experts entity representation framework MoMoK for knowledge graph completion task. MoMoK consists of three different modules called relation-guided modality knowledge experts, multi-modal joint decision, and expert information disentanglement.
weaknesses: - The discussion of MAIN RESULTS needs to go further, have the authors explored in depth why MKG-Y has a lesser MRR than baseline AdaMF?
- The fonts of the figure3 and figure4 are not consistent.
- The detailed settings of Section5.4 should be more carefully introduced. About how the training datasets are built and the detailed evaluation protocol.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Given that the expert networks rely on raw embeddings, how does the model ensure that sufficient diversity within each modality is captured? Are there mechanisms in place to mitigate the potential loss of key details due to the use of compressed embeddings?
2. Could you provide further clarification on the rationale behind using the conditional probabilities of different entities for disentanglement in formula 8? How does this approach ensure effective disentanglement between experts?
3. In the final case study, the authors visualize the weighting between modalities. How does the model distinguish and prioritize features within a single modality?
soundness: 3
strengths: 1. The introduction of relation-guided multimodal expert networks with adaptive weighting mechanisms enables the model to fully exploit different perspectives within each modality based on relational context, which represents a novel and impactful focus.
2. By minimizing mutual information, the model effectively reduces redundant information between modalities, enhancing disentanglement among the experts. This is a particularly noteworthy design feature.
3. The ablation experiments validate the importance of each module in the model's performance, particularly highlighting the significance of joint training and information disentanglement, proving the rationale behind the design.
4. The paper presents five well-defined research questions (RQs) that thoroughly address the model’s strengths, performance in complex environments, and the contribution of each design module. This makes the experimental section well-structured and logically cohesive.
summary: This paper introduces a novel model named MOMOK (Mixture of Modality Knowledge Experts), designed to address the challenges in multimodal knowledge graph completion tasks. The innovation lies in the relation-guided multimodal expert networks, which dynamically adjust the weighting of modal information based on relation, enhancing the model's completion performance. The experimental results demonstrate that MOMOK surpasses existing baseline models across multiple public datasets, showcasing its effectiveness in handling multimodal information. The paper is well-writing with contributions in both theoretical design and empirical results.
weaknesses: 1. The novelty is somewhat limited. The paper claims existing works lack considering the relational context, and proposes MoMoK frameworks by incorporates relation-guided modality knowledge experts for each modality, which constructs expert networks in each modality. But in my opinion, there are also lots of prior work (such as MoSE) considering relational context. 
2. It would be beneficial to introduce datasets used in baselines for comprehensive comparison. The baselines, such as MoSE, further conduct experiments on (FB-15K, WN18, WN9) which this paper did not used. Including these datasets would be helpful to further demonstrate the robustness of the proposed model.
3. The expert networks rely on raw embeddings, which may limit their ability to fully capture the diversity within each modality. This could lead to less distinct perspectives being generated by the experts, especially if the raw embeddings have already compressed or lost key details, thereby reducing the overall effectiveness of the expert network design.
4. While the paper introduces CLUB to estimate the upper bound of mutual information and designs a corresponding loss function, it lacks sufficient explanation for why the conditional probabilities of different entities are used for disentanglement in formula 8.


Other Minor Weaknesses:

1. The related work section is too brief and fails to emphasize the superiority of the proposed framework compared to existing approaches. For example, in the experimental section, you mentioned the similarities between your method and MoSE and IMF as baselines, but you did not provide a more detailed discussion of the differences. This could be elaborated in the related work section to clarify the distinctions between your approach and these existing methods.
2. The paper discusses the use of three modalities, image, text, and structure
but it appears to lack a clear explanation regarding how the raw embeddings for the structural modality are obtained. Neither the main body of the paper nor the appendix provides detailed information about the specific model or method used to generate raw embeddings for the structural modality.

Typos:

1. Page2 'constrative' maybe is 'contrastive'
2. The second-best method in the MRR indicator of the MKG-W dataset in Table 2 should be MMRNS (35.03), but it was incorrectly marked as IMF (34.50).

---

confidence: 4
contribution: 2
initial_rating: 8
presentation: 4
questions: 1. In the second paragraph of Section 4.1, are $\mathcal{W}\_{m,1}$, $\mathcal{W}\_{m,2}$, ... learnable matrics? What are their shapes? Also, what is the exact operation in $\mathcal{V}\_{m,i}^{e}=\mathcal{W}\_{m,i}(e\_m)$? If it is just a simple multiplication, I feel using parentheses is improper. 
2. In Eq. (2), what is the purpose/function of the temperature $\varepsilon_r$?
3. In Eq. (3), I think the relation $r$ is missing in the components $P\_m(\hat{e}\_{m})$ and $P\_n(\hat{e}\_{n})$?
4. In line 266, where the joint score of each triple is treated as the sum of the scores across all modalities, did the authors try other designs, such as using only the '$J$' modality score? I am curious about the results of using only the '$J$' modality.
5. What's the setting of the number of experts (the value of $K$) in the main experiment? This should be described in the implementation details.
6. In Section 5.1, what is the difference between the MKG-W and MKG-Y datasets?
soundness: 3
strengths: The paper is well-written, and I was able to grasp authors' motivation and the general process of MoMoK at first glance. The algorithm is clearly presented overall. The performance of MoMoK surpassed existing methods by a significant margin. Sufficient experimental evidence supports the robustness of MoMoK.
summary: The authors argue the limitations of entity-wise embedding fusion in existing MMKGC work and introduce a new method, MoMoK, which can learn relation-guided entity representations for each modality. The experimental results demonstrate the superiority and robustness of MoMoK.
weaknesses: I think **the discussion of the importance of KG Completion task, especially its position in the context of the pervasiveness of LLMs today, is lacking**. This reduces the overall interest and significance of the work from the audience's perspective, considering LLMs' strong performance in general knowledge reasoning. I suggest the authors add a brief discussion in the introduction or in the last section. Additionally, several places lack detail and need clarification (see questions).

Meta-review: This paper proposes a multi-modal knowledge graph (MMKG) representation learning model, a Mixture of Modality Knowledge experts (MoMoK for short), to learn adaptive multi-modal entity representations for better MMKG completion. A key design idea is to learn representations through relation guidance and expert information in decoupling MMKGs. The method exploits a relation-guided modal knowledge expert network, adaptively aggregating multi-view embeddings and adopting a multimodal joint decision mechanism.

The initial reviews raised concerns about missing metrics/datasets/ablation studies, limited novelty, and unclear advantages or details. The authors have worked to improve the quality of the paper during the rebuttal and discussion period by adding more experiments (e.g., showing additional metrics and conducting additional ablation studies) and providing more details and analyses. Some reviewers acknowledged this improvement. These additional experimental results and explanations should be reflected in the final version of the paper.

Decision: Accept (Poster)