Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning

Authors: Yichi Zhang, Zhuo Chen, Lingbing Guo, yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, Huajun Chen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multi-modal Information Fusion, Knowledge Graph, Multi-modal Entity Representation, Mixture-of-Experts

Abstract:
Learning high-quality multi-modal entity representations is an important goal of multi-modal knowledge graph (MMKG) representation learning, which can en- hance reasoning tasks within the MMKGs, such as MMKG completion (MMKGC). The main challenge is to collaboratively model the structural information concealed in massive triples and the multi-modal features of the entities. Existing methods focus on crafting elegant entity-wise multi-modal fusion strategies, yet they over- look the utilization of multi-perspective features concealed within the modalities under diverse relational contexts. To address this issue, we introduce a novel framework with Mixture of Modality Knowledge experts (MOMOK for short) to learn adaptive multi-modal entity representations for better MMKGC. We design relation-guided modality knowledge experts to acquire relation-aware modality embeddings and integrate the predictions from multi-modalities to achieve joint decisions. Additionally, we disentangle the experts by minimizing their mutual information. Experiments on four public MMKG benchmarks demonstrate the outstanding performance of MOMOK under complex scenarios. Our code and data are available at https://github.com/zjukg/MoMoK.

Decision: accept (poster)
Reviewer scores: 5, 6, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1729462219281

thread_type:
official_review

<thread_object>
creator: reviewer_mwpc

<content>
summary:
The authors argue the limitations of entity-wise embedding fusion in existing MMKGC work and introduce a new method, MoMoK, which can learn relation-guided entity representations for each modality. The experimental results demonstrate the superiority and robustness of MoMoK.

strengths:
The paper is well-written, and I was able to grasp authors' motivation and the general process of MoMoK at first glance. The algorithm is clearly presented overall. The performance of MoMoK surpassed existing methods by a significant margin. Sufficient experimental evidence supports the robustness of MoMoK.

weaknesses:
I think **the discussion of the importance of KG Completion task, especially its position in the context of the pervasiveness of LLMs today, is lacking**. This reduces the overall interest and significance of the work from the audience's perspective, considering LLMs' strong performance in general knowledge reasoning. I suggest the authors add a brief discussion in the introduction or in the last section. Additionally, several places lack detail and need clarification (see questions).

questions:
1. In the second paragraph of Section 4.1, are $\mathcal{W}\_{m,1}$, $\mathcal{W}\_{m,2}$, ... learnable matrics? What are their shapes? Also, what is the exact operation in $\mathcal{V}\_{m,i}^{e}=\mathcal{W}\_{m,i}(e\_m)$? If it is just a simple multiplication, I feel using parentheses is improper. 
2. In Eq. (2), what is the purpose/function of the temperature $\varepsilon_r$?
3. In Eq. (3), I think the relation $r$ is missing in the components $P\_m(\hat{e}\_{m})$ and $P\_n(\hat{e}\_{n})$?
4. In line 266, where the joint score of each triple is treated as the sum of the scores across all modalities, did the authors try other designs, such as using only the '$J$' modality score? I am curious about the results of using only the '$J$' modality.
5. What's the setting of the number of experts (the value of $K$) in the main experiment? This should be described in the implementation details.
6. In Section 5.1, what is the difference between the MKG-W and MKG-Y datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704931609

thread_type:
official_review

<thread_object>
creator: reviewer_rudt

<content>
summary:
This paper introduces a novel model named MOMOK (Mixture of Modality Knowledge Experts), designed to address the challenges in multimodal knowledge graph completion tasks. The innovation lies in the relation-guided multimodal expert networks, which dynamically adjust the weighting of modal information based on relation, enhancing the model's completion performance. The experimental results demonstrate that MOMOK surpasses existing baseline models across multiple public datasets, showcasing its effectiveness in handling multimodal information. The paper is well-writing with contributions in both theoretical design and empirical results.

strengths:
1. The introduction of relation-guided multimodal expert networks with adaptive weighting mechanisms enables the model to fully exploit different perspectives within each modality based on relational context, which represents a novel and impactful focus.
2. By minimizing mutual information, the model effectively reduces redundant information between modalities, enhancing disentanglement among the experts. This is a particularly noteworthy design feature.
3. The ablation experiments validate the importance of each module in the model's performance, particularly highlighting the significance of joint training and information disentanglement, proving the rationale behind the design.
4. The paper presents five well-defined research questions (RQs) that thoroughly address the model’s strengths, performance in complex environments, and the contribution of each design module. This makes the experimental section well-structured and logically cohesive.

weaknesses:
1. The novelty is somewhat limited. The paper claims existing works lack considering the relational context, and proposes MoMoK frameworks by incorporates relation-guided modality knowledge experts for each modality, which constructs expert networks in each modality. But in my opinion, there are also lots of prior work (such as MoSE) considering relational context. 
2. It would be beneficial to introduce datasets used in baselines for comprehensive comparison. The baselines, such as MoSE, further conduct experiments on (FB-15K, WN18, WN9) which this paper did not used. Including these datasets would be helpful to further demonstrate the robustness of the proposed model.
3. The expert networks rely on raw embeddings, which may limit their ability to fully capture the diversity within each modality. This could lead to less distinct perspectives being generated by the experts, especially if the raw embeddings have already compressed or lost key details, thereby reducing the overall effectiveness of the expert network design.
4. While the paper introduces CLUB to estimate the upper bound of mutual information and designs a corresponding loss function, it lacks sufficient explanation for why the conditional probabilities of different entities are used for disentanglement in formula 8.


Other Minor Weaknesses:

1. The related work section is too brief and fails to emphasize the superiority of the proposed framework compared to existing approaches. For example, in the experimental section, you mentioned the similarities between your method and MoSE and IMF as baselines, but you did not provide a more detailed discussion of the differences. This could be elaborated in the related work section to clarify the distinctions between your approach and these existing methods.
2. The paper discusses the use of three modalities, image, text, and structure
but it appears to lack a clear explanation regarding how the raw embeddings for the structural modality are obtained. Neither the main body of the paper nor the appendix provides detailed information about the specific model or method used to generate raw embeddings for the structural modality.

Typos:

1. Page2 'constrative' maybe is 'contrastive'
2. The second-best method in the MRR indicator of the MKG-W dataset in Table 2 should be MMRNS (35.03), but it was incorrectly marked as IMF (34.50).

questions:
1. Given that the expert networks rely on raw embeddings, how does the model ensure that sufficient diversity within each modality is captured? Are there mechanisms in place to mitigate the potential loss of key details due to the use of compressed embeddings?
2. Could you provide further clarification on the rationale behind using the conditional probabilities of different entities for disentanglement in formula 8? How does this approach ensure effective disentanglement between experts?
3. In the final case study, the authors visualize the weighting between modalities. How does the model distinguish and prioritize features within a single modality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712216670

thread_type:
official_review

<thread_object>
creator: reviewer_diph

<content>
summary:
This paper proposes a multi-modal mixture-of-experts entity representation framework MoMoK for knowledge graph completion task. MoMoK consists of three different modules called relation-guided modality knowledge experts, multi-modal joint decision, and expert information disentanglement.

strengths:
- Multi-modal knowledge graph representation learning is a hot topic in the KG community.
- The three modules in the design of the method proposed by the authors are self-consistent and the authors provide a more detailed theoretical proof.
- The experimental part of the workload is sufficient, and the authors have done more experiments to prove the effectiveness, robustness, and efficiency of the method. Detailed case study and visualization results are also provided.

weaknesses:
- The discussion of MAIN RESULTS needs to go further, have the authors explored in depth why MKG-Y has a lesser MRR than baseline AdaMF?
- The fonts of the figure3 and figure4 are not consistent.
- The detailed settings of Section5.4 should be more carefully introduced. About how the training datasets are built and the detailed evaluation protocol.

questions:
- Please make a response to the comments in weaknesses.
- What exactly are the advantages of the traditional embedding-based approach over the LLM-based approach for knowledge graph completion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730857168443

thread_type:
official_review

<thread_object>
creator: reviewer_mptp

<content>
summary:
This paper proposes a novel framework, Mixture of Modality Knowledge Experts (MoMoK), which aims to improve the quality of multimodal entity representation learning through relation guidance and expert information decoupling in multimodal knowledge graphs (MMKGs). MoMoK achieves better triple prediction by designing a relation-guided modal knowledge expert network, adaptively aggregating multi-view embeddings, and adopting a multimodal joint decision mechanism. Experimental results show that MoMoK performs well on four public MMKG benchmarks and significantly outperforms existing methods.

strengths:
1. This paper considers the modal information hidden in different relational contexts and designs a relation-guided modal knowledge expert network to adaptively aggregate multi-view embeddings.
2. On four public MMKG benchmarks, MoMoK significantly outperforms recent baseline methods and demonstrates its superior performance in complex scenarios.
3. Through detailed ablation analysis, this paper verifies the effectiveness of the design and provides intuitive case studies to enhance the interpretability of the model.

weaknesses:
1. Table 2 lacks a comparison of the number of model parameters. Although the authors have discussed the training memory consumption and time efficiency of some methods in Section 5.6, it is beneficial to supplement these values for all baseline models in Table 2 for fair comparison.
2. The description in Table 4 does not seem to be that comprehensive. Does the GPU memory here refer to the memory during inference or the memory during training? Does the time in seconds represent the time it takes to infer one sample?

questions:
1. The parameter \lambda seems small enough. Can you explain why club loss can improve the model by about one point in almost every metric under a weight of 1e-4 (Table 3)?
2. The method seems to be insufficiently integrated with the multimodal large model. This paper uses BERT and VGG to extract the original features of the model. Have you tried to use a multimodal large model to represent them uniformly? I would like to know the performance comparison between the embedding representation of the multimodal large model and MoMoK.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730949254521

thread_type:
official_review

<thread_object>
creator: reviewer_vfnt

<content>
summary:
This paper tackles MKGE problems by proposing a Mixture of Modality Knowledge Experts framework. The authors develop relation-guided modality knowledge experts to obtain relation-aware modality embeddings while reducing mutual information among the experts. Empirical studies validate the model's effectiveness.

strengths:
1. Experiments in complex environments demonstrate MoMoK's resilience to noise, data sparsity, and missing modalities.  
2. This work introduces a novel expert information disentanglement (ExID) module to separate expert decisions for each modality.  
3. The rationale for designing the relation-aware MKGC method is evident.

weaknesses:
1. Metrics Reporting: Table 2 displays only the Hit@1 and MRR metrics for MKG-W and MKG-Y, showing modest performance improvements for MoMoK. Please include additional metrics for MoMoK, AdaMF, and MMRNS to better assess the reported gains.

2. MoMoK on Classic MMKGC Datasets: Why wasn't MoMoK evaluated on classic MMKGC datasets like WN9-IMG, FB-IMG, WN18-IMG, and FB15K-237-IMG?

3. Modal Data and Encoder Details: The notation $\mathcal{X}_m(e)$ indicates information from multiple modalities, yet four datasets are used. Please clarify the specific modalities for each dataset, how multiple images are handled, and whether VGG and BERT are consistently used as encoders.

4. Notation Standardization: Some formulas, such as $\mathcal{E}_h$ in formula (1), could benefit from standardized notation to improve clarity and consistency.

5. Relation-Aware Temperature ($\mathcal{E}_r$): The relation-aware temperature $\mathcal{E}_r$ is important for GFN computation, but its definition is absent. Please provide its formal definition and its impact on model performance.

6. Score Function Comparison: The proposed method employs Tucker decomposition as its score function, while the baseline methods utilize various other functions.

7. Ablation Study: Ablation experiments indicate that components like adaptive fusion, noise $\delta_{r}$, and relational $\epsilon_{m}$ may have minimal individual impacts. Further analysis or emphasis on their significance in different experimental contexts is needed.

8. Ablation Experiments for MKG-Y and KVC16K: Can MKG-Y and KVC16K datasets be included in the ablation experiments? This would enhance the assessment of the method's performance across diverse data and conditions.

questions:
please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731901040263

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Gentle Message for ICLR Rebuttal from Authors

comment:
Dear PCs, SACs, ACs, and Reviewers:

After many days of hard work, we have finished **responding to each question in the review comments one by one**, and for some questions, we have added the corresponding experimental results. We hope you can see our Rebuttal responses and give us your feedback. We hope that our responses can solve your problems and improve your impression of our paper.

Sincerely,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732159820010

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Gentle Request for Rebuttal

comment:
Dear Reviewers:

As the discussion deadline getting closer, we gently hope that you can participate in the rebuttal and give some possible feedback on our rebuttal responses.

Sincerely,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732350707308

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Gentle Request for Rebuttal

comment:
Dear Reviewers:

As the rebuttal period would be end. We really hope that you can read our response and give us some further feedback.

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a multi-modal knowledge graph (MMKG) representation learning model, a Mixture of Modality Knowledge experts (MoMoK for short), to learn adaptive multi-modal entity representations for better MMKG completion. A key design idea is to learn representations through relation guidance and expert information in decoupling MMKGs. The method exploits a relation-guided modal knowledge expert network, adaptively aggregating multi-view embeddings and adopting a multimodal joint decision mechanism.

The initial reviews raised concerns about missing metrics/datasets/ablation studies, limited novelty, and unclear advantages or details. The authors have worked to improve the quality of the paper during the rebuttal and discussion period by adding more experiments (e.g., showing additional metrics and conducting additional ablation studies) and providing more details and analyses. Some reviewers acknowledged this improvement. These additional experimental results and explanations should be reflected in the final version of the paper.