Multimodal Analogical Reasoning over Knowledge Graphs
Authors: Ningyu Zhang
Lei Li
Xiang Chen
Xiaozhuan Liang
Shumin Deng
Huajun Chen
Venue: ICLR 2023 poster (2023)
Keywords: knowledge graph
multimodal
analogical reasoning
prompt learning
pre-trained language model

Abstract: Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance. We hope our work can deliver benefits and inspire future research. Code and datasets are available in https://github.com/zjunlp/MKG_Analogy.


Reviews:
[{"review_id": "kpaNECoMw-e", "reviewer": "Reviewer_R5x6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new task of multimodal analogical reasoning that takes the inputs of a pair of head and tail entities in different modalities, and predicts the missing tail entity for a given query head entity. The task is inspired by conventional single modal analogical reasoning and cognitive theory: human learns well from multiple modal sources instead of a single one. To build the benchmark, the authors collect images from Google for textual entities in KG, and conduct several pre-processing steps to clean the data. Furthermore, three baseline methods are proposed, including MKGE methods and pre-training methods.\n", "summary_of_the_review": "see above", "strength_and_weaknesses": "Strengths:\n1. The motivation is reasonable and the paper is mostly clear.\n2. The paper contributes a new task and dataset for multimodal reasoning.\n\nWeaknesses:\n\n1. The presentation needs further improvements. For example, 1) fig. 3 and Section 4.2.1 is not clear. What exactly are the e_h and e_t in the inputs? Why are there two [PAD]? Are there two inputs or just one? What is the [MASK] for? why the [MASK] needs to multiply E_e1 and E_ri? 2) How to ensure the quality of the proposed dataset? Even though I have checked the Appendix, I still cannot find the annotation guideline or agreement among annotators.\n\n2. The experiments need further discussion to show the necessity of the task and the quality of the dataset. 1) The multi-modal analogical reasoning task can be decomposed into two sub tasks: cross-modal alignment and single-modal analogical reasoning. What is the advantage of evaluating them at the same time? 2) What is the main reason for the unsatisfactory performance, low-quality data? difficult cross-modal alignment or analogical reasoning? I would suggest to evaluate the two sub-tasks separately for further investigation. 3) From the cases in fig. 6, the predicted entities varies a lot. Why?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OuYMGQU73L", "reviewer": "Reviewer_Hqif", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new dataset and knowledge graph for multimodal analogical reasoning. \nThe knowledge graph (MarKG) is based on E-KAR and BATs, augmented by Wikidata and image search results from Google and Laion-5B. The dataset (MARS) is based on analogy relations in E-KAR and BATs; the task is that given two entities (e_h, e_t) with hidden relation r, and a question entity e_q, to predict the entity e_a with the same relation r to e_q. The given entities can be in an either textual or visual format. \n\nThe paper also proposes several methods for multimodal analogical reasoning (MAR). The first method uses existing multimodal entity embedding methods, with the ANALOGY loss from previous work. The second method uses a pretrained multimodal transformer (MPT). The authors first pretrain the MPT over MarKG by linearizing the KG; then they propose two modifications. The first one is multiplying the attention probability between head and tail entities by a learned parameter. The second is an additional loss that brings the contextual relation representation closer and entity representation further. \n\nThe results show that the proposed methods can improve the performance over naïve MPTs. MKGE methods perform competitively on this task, and only MKGformer can outperform the MKGE methods among the 5 MPTs that the authors tried.", "summary_of_the_review": "In all, I feel the paper has good contributions, but the quality is preventing me from giving a higher score. ", "strength_and_weaknesses": "Strength: The paper gives a novel KG along with the dataset for MAR. The authors also propose a method for MAR with transformers and show solid improvement over 5 different backbones.\n\nWeakness: \n\n1. The methods proposed for MAR are not very novel, as it is quite common among existing KG and language model methods. \n2. Essentially, the proposed dataset is just an augmentation of E-KAR and BATs with images - the analogy data is already existing and the new things are just the images. If I understand correctly, MARS does not contain any data from Wikidata, and they do not have any tasks on MarKG. Therefore the wikidata part in MarKG is just for training purposes.\n3. I think a human evaluation is essential for the MARS dataset. We need to know how human performs on this task to get an idea of the data quality.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CKbin5ivsMg", "reviewer": "Reviewer_JzuQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**High level motivation:** While the multimodality of human perception is shown to help with analogical reasoning, there isn't a standardized task adopted by the machine learning community to drive progress in this area. \n\n**Contributions:**\n* **New task and dataset:** The authors propose a new multimodal analogical reasoning task over knowledge graphs. They construct a dataset named MARS (Multimodal Analogical Reasoning dataSet) to evaluate the performance in this task, and a knowledge graph named MarKG to support pretraining. \n  * This contribution subsumes details regarding how the inputs/outputs should be formatted/prompted during pretraining and finetuning. Given that there are likely multiple ways to do this, this is a nontrivial contribution (I believe the authors' choices are sensible)\n* **Extensive evaluation of existing benchmarks:** The authors run an extensive benchmarking evaluations and ablations to identify how the current SOTA approaches in analogical reasoning perform. They reach nontrivial conclusions, such as certain architectural components (especially the ANALOGY backbone by Liu et. al. (2017)). \n* **Architectural and training modifications to boost performance:** The authors propose MarT, a framework that consists of a modification to the attention mechanism of a transformer and an added loss to support Structure Mapping from cognitive science. Their proposed modifications seem to lead to improvements in their ablations. \n* **Insightful discussion:** The authors end the paper with an insightful discussion of the findings. Some highlights include the models' ability to generalize to novel relations (and hte effect of pretraining on this) and the role of the proposed relaxation loss. ", "summary_of_the_review": "This is a strong submission with clear contributions (unless there's omitted prior work that I'm not aware of). \n\nI believe that the research community would be better off with this paper accepted to ICLR. ", "strength_and_weaknesses": "**STRENGHT:**\n* **Strong motivation:** Given the increasing interest in multimodal reasoning, I believe that this paper is filling an important gap in literature. Just the dataset and task definition alone constitute strong contributions. \n* **Multiple strong contributions:** The benchmarking and ablation experiments are comprehensive enough that they alone (i.e. excluding the dataset) could perhaps constitute a separate submission. \n* **Dataset and test definition sensible:** Both the way the dataset is constructed (about which I'll list some clarifying questions below), the input and outputs are formatted and the way the models are evaluated (pass@k) seem quite sensible, and likely can be used without much modification by other researchers interested in multimodal analogical reasoning. \n\n**WEAKNESSES:**\n* **\"No input pairs\" baseline is missing:** It would be great if the authors evaluated the performance of the models when they are trained **without** with the analogy pair to condition on. That is, instead of following the current format, which is \"if x is to y, then z is to ?', drop the first bit and only train and test on \"z is to ?\". This would give us the base rates and the performance obtained by this model would be very useful to make sense of how high the reported accuracies are. \n* **Missing scaling plots:** While providing scaling plots is not standard practice today (which is why this critique should not be held strongly against the authors), I believe they provide priceless information that's hard to substitute. For example, I remain skeptical that the modification introduced in the self-attention equations (the learned gating) will continue to matter at larger model scales. Scaling plots can serve to dispel these suspicions. \n* **No baseline without explicitly inferring relations:** While the choice to stick to the \"abduct-relate-infer\" pipeline is sensible, it still remains to be seen whether DL models can still perform well without this explicit structure. The submission would be stronger if the authors at least reported some preliminary results on the effect of relation inference. \n* **(Nitpick) Multi-pair instances:** This is not really a weakness, but perhaps a sensible next step(?): It seems relatively easy to convert this dataset into a k-shot analogy reasoning one. That is, instead of a single analogy pair, there could be k number of them. Perhaps this could help authors reach new conclusions that cannot be reached using the current version of the dataset. \n* **A bit confusing sentence:** What exactly do you mean by this: \"The structure mapping theory described that relations\nbetween objects, rather than attributes of objects, are mopped from base to target.\"\n* **(nitpick)** Results in Table 2 don't **prove** Mayer's theory, but support it. Just wanted to highlight this!\n\n\n\n\n**QUESTIONS TO AUTHORS:**\n* **Entity filtering:** You mention in the Appendix that you constrain the entities to be visualizable. How did you enforce this? Also what does the following mean exactly? \"the relation must imply analogical knowledge reasoning rather than simple word linear analogy\"\n* **Removing corrupted and low quality iamges:** How does this step, also outlined in appendix, work exactly?\n* **MarKG vs. MARS:** Could you precisely describe what the connection between these is? How much of an overlap is there between these?\n* **Text description vs. entity token:** This is perhaps a big misunderstanding, but I didn't quite get the distinction between what the difference between a \"text description\" and an \"entity token\" is. \n* **prompt tuning:** What exactly do you mean by \"prompt tuning\" in Remark 1? Prompt tuning is a method to train the context embeddings by backpropping on them using a given task loss. I presume you don't mean this? \n* **Gates in MarT:** If these gates are learnable parameters, how do they adaptively adjust how much cross-entity attention happens?\n* **Position embeddings:** What position (if any) do you use? The gates in MarT resemble a more restricted version of T5-style position encodings to me. Is this right? \n* **Noise data:** Just wanted to clarify whether you mean \"noisy data\" in Remark 2? In either case, I don' quite get this point. \n* **Generalizing to novel relations:** 1) What about the MarKG is helping the models acquire this capability? 2) You have some ablations without MarKG pretraining that still obtain nontrivial accuracy. How is this possible, given that the model has never interacted with a novel relation yet (even the embeddings for that relation is, as far as I understand, won't be learned)? More on this would be great!\n* **Hyperparameter tuning strategy:** Could you outline the hyperparameter tuning strategies you used? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster