Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **High level motivation:** While the multimodality of human perception is shown to help with analogical reasoning, there isn't a standardized task adopted by the machine learning community to drive progress in this area. 

**Contributions:**
* **New task and dataset:** The authors propose a new multimodal analogical reasoning task over knowledge graphs. They construct a dataset named MARS (Multimodal Analogical Reasoning dataSet) to evaluate the performance in this task, and a knowledge graph named MarKG to support pretraining. 
  * This contribution subsumes details regarding how the inputs/outputs should be formatted/prompted during pretraining and finetuning. Given that there are likely multiple ways to do this, this is a nontrivial contribution (I believe the authors' choices are sensible)
* **Extensive evaluation of existing benchmarks:** The authors run an extensive benchmarking evaluations and ablations to identify how the current SOTA approaches in analogical reasoning perform. They reach nontrivial conclusions, such as certain architectural components (especially the ANALOGY backbone by Liu et. al. (2017)). 
* **Architectural and training modifications to boost performance:** The authors propose MarT, a framework that consists of a modification to the attention mechanism of a transformer and an added loss to support Structure Mapping from cognitive science. Their proposed modifications seem to lead to improvements in their ablations. 
* **Insightful discussion:** The authors end the paper with an insightful discussion of the findings. Some highlights include the models' ability to generalize to novel relations (and hte effect of pretraining on this) and the role of the proposed relaxation loss. 
strength_and_weaknesses: **STRENGHT:**
* **Strong motivation:** Given the increasing interest in multimodal reasoning, I believe that this paper is filling an important gap in literature. Just the dataset and task definition alone constitute strong contributions. 
* **Multiple strong contributions:** The benchmarking and ablation experiments are comprehensive enough that they alone (i.e. excluding the dataset) could perhaps constitute a separate submission. 
* **Dataset and test definition sensible:** Both the way the dataset is constructed (about which I'll list some clarifying questions below), the input and outputs are formatted and the way the models are evaluated (pass@k) seem quite sensible, and likely can be used without much modification by other researchers interested in multimodal analogical reasoning. 

**WEAKNESSES:**
* **"No input pairs" baseline is missing:** It would be great if the authors evaluated the performance of the models when they are trained **without** with the analogy pair to condition on. That is, instead of following the current format, which is "if x is to y, then z is to ?', drop the first bit and only train and test on "z is to ?". This would give us the base rates and the performance obtained by this model would be very useful to make sense of how high the reported accuracies are. 
* **Missing scaling plots:** While providing scaling plots is not standard practice today (which is why this critique should not be held strongly against the authors), I believe they provide priceless information that's hard to substitute. For example, I remain skeptical that the modification introduced in the self-attention equations (the learned gating) will continue to matter at larger model scales. Scaling plots can serve to dispel these suspicions. 
* **No baseline without explicitly inferring relations:** While the choice to stick to the "abduct-relate-infer" pipeline is sensible, it still remains to be seen whether DL models can still perform well without this explicit structure. The submission would be stronger if the authors at least reported some preliminary results on the effect of relation inference. 
* **(Nitpick) Multi-pair instances:** This is not really a weakness, but perhaps a sensible next step(?): It seems relatively easy to convert this dataset into a k-shot analogy reasoning one. That is, instead of a single analogy pair, there could be k number of them. Perhaps this could help authors reach new conclusions that cannot be reached using the current version of the dataset. 
* **A bit confusing sentence:** What exactly do you mean by this: "The structure mapping theory described that relations
between objects, rather than attributes of objects, are mopped from base to target."
* **(nitpick)** Results in Table 2 don't **prove** Mayer's theory, but support it. Just wanted to highlight this!




**QUESTIONS TO AUTHORS:**
* **Entity filtering:** You mention in the Appendix that you constrain the entities to be visualizable. How did you enforce this? Also what does the following mean exactly? "the relation must imply analogical knowledge reasoning rather than simple word linear analogy"
* **Removing corrupted and low quality iamges:** How does this step, also outlined in appendix, work exactly?
* **MarKG vs. MARS:** Could you precisely describe what the connection between these is? How much of an overlap is there between these?
* **Text description vs. entity token:** This is perhaps a big misunderstanding, but I didn't quite get the distinction between what the difference between a "text description" and an "entity token" is. 
* **prompt tuning:** What exactly do you mean by "prompt tuning" in Remark 1? Prompt tuning is a method to train the context embeddings by backpropping on them using a given task loss. I presume you don't mean this? 
* **Gates in MarT:** If these gates are learnable parameters, how do they adaptively adjust how much cross-entity attention happens?
* **Position embeddings:** What position (if any) do you use? The gates in MarT resemble a more restricted version of T5-style position encodings to me. Is this right? 
* **Noise data:** Just wanted to clarify whether you mean "noisy data" in Remark 2? In either case, I don' quite get this point. 
* **Generalizing to novel relations:** 1) What about the MarKG is helping the models acquire this capability? 2) You have some ablations without MarKG pretraining that still obtain nontrivial accuracy. How is this possible, given that the model has never interacted with a novel relation yet (even the embeddings for that relation is, as far as I understand, won't be learned)? More on this would be great!
* **Hyperparameter tuning strategy:** Could you outline the hyperparameter tuning strategies you used? 

clarity,_quality,_novelty_and_reproducibility: **Clarity:**
* While there are numerous grammar issues in the text, the text still flows well and is quite readable. this was **not** used against the authors in the review. 
* Precisely what the inputs to the neural networks are is not extremely clear. Notation and descriptions could be improved to remove any doubt. 
* Figure 4 is a bit hard to understand. Perhaps it's better to make it understandable standalone (i.e. it makes sense without referring to the text). 
* Figure 3a: Why is the answer "Young Tesla", not just "Tesla"?

**Quality:** I believe that this submission is high quality, regarding the professionality of the writing/figures and the technical content. 

**Novelty:** To the extent of my knowledge, the task, datasets, evaluations and the contents of the discussion are all sufficiently novel. 

**Reproducibility:** While the paper could describe the model tuning procedure a bit more, all the results can likely be verified thanks to the dataset and code release. (I have not checked the code)
summary_of_the_review: This is a strong submission with clear contributions (unless there's omitted prior work that I'm not aware of). 

I believe that the research community would be better off with this paper accepted to ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new dataset and knowledge graph for multimodal analogical reasoning. 
The knowledge graph (MarKG) is based on E-KAR and BATs, augmented by Wikidata and image search results from Google and Laion-5B. The dataset (MARS) is based on analogy relations in E-KAR and BATs; the task is that given two entities (e_h, e_t) with hidden relation r, and a question entity e_q, to predict the entity e_a with the same relation r to e_q. The given entities can be in an either textual or visual format. 

The paper also proposes several methods for multimodal analogical reasoning (MAR). The first method uses existing multimodal entity embedding methods, with the ANALOGY loss from previous work. The second method uses a pretrained multimodal transformer (MPT). The authors first pretrain the MPT over MarKG by linearizing the KG; then they propose two modifications. The first one is multiplying the attention probability between head and tail entities by a learned parameter. The second is an additional loss that brings the contextual relation representation closer and entity representation further. 

The results show that the proposed methods can improve the performance over naïve MPTs. MKGE methods perform competitively on this task, and only MKGformer can outperform the MKGE methods among the 5 MPTs that the authors tried.
strength_and_weaknesses: Strength: The paper gives a novel KG along with the dataset for MAR. The authors also propose a method for MAR with transformers and show solid improvement over 5 different backbones.

Weakness: 

1. The methods proposed for MAR are not very novel, as it is quite common among existing KG and language model methods. 
2. Essentially, the proposed dataset is just an augmentation of E-KAR and BATs with images - the analogy data is already existing and the new things are just the images. If I understand correctly, MARS does not contain any data from Wikidata, and they do not have any tasks on MarKG. Therefore the wikidata part in MarKG is just for training purposes.
3. I think a human evaluation is essential for the MARS dataset. We need to know how human performs on this task to get an idea of the data quality.


clarity,_quality,_novelty_and_reproducibility: The paper is not very well written in my opinion. There are various typos and it is sometimes hard to understand.

I still have some questions:
1. What are e_h, e_t, and e_q in (1)? Are they special tokens? This is not mentioned in the paper. 4.2.1 mentioned that all the entities and relations are special tokens, but for a question, we do not know the ground truth entity id for I_h and I_t.
2. Appendix B.1, Step 1: how do you perform the filtering of entities and relations? Did you check them manually? The criteria is vague.
3. B.1 Step 3: Did you get 5 images for all the 11k entities? Are all the entities visualizable? I wonder if some entities are quite abstract to represent with an image.

Typos:
1. Sec 4.3, adaptive Interaction: It should be "noisy data" instead of "noise data".
2. Sec 4.3, relation oriented: are "mapped" from, instead of are "mopped" from.

The paper proposes novel ideas for MAR. Codes and data are included with the submission.
summary_of_the_review: In all, I feel the paper has good contributions, but the quality is preventing me from giving a higher score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new task of multimodal analogical reasoning that takes the inputs of a pair of head and tail entities in different modalities, and predicts the missing tail entity for a given query head entity. The task is inspired by conventional single modal analogical reasoning and cognitive theory: human learns well from multiple modal sources instead of a single one. To build the benchmark, the authors collect images from Google for textual entities in KG, and conduct several pre-processing steps to clean the data. Furthermore, three baseline methods are proposed, including MKGE methods and pre-training methods.

strength_and_weaknesses: Strengths:
1. The motivation is reasonable and the paper is mostly clear.
2. The paper contributes a new task and dataset for multimodal reasoning.

Weaknesses:

1. The presentation needs further improvements. For example, 1) fig. 3 and Section 4.2.1 is not clear. What exactly are the e_h and e_t in the inputs? Why are there two [PAD]? Are there two inputs or just one? What is the [MASK] for? why the [MASK] needs to multiply E_e1 and E_ri? 2) How to ensure the quality of the proposed dataset? Even though I have checked the Appendix, I still cannot find the annotation guideline or agreement among annotators.

2. The experiments need further discussion to show the necessity of the task and the quality of the dataset. 1) The multi-modal analogical reasoning task can be decomposed into two sub tasks: cross-modal alignment and single-modal analogical reasoning. What is the advantage of evaluating them at the same time? 2) What is the main reason for the unsatisfactory performance, low-quality data? difficult cross-modal alignment or analogical reasoning? I would suggest to evaluate the two sub-tasks separately for further investigation. 3) From the cases in fig. 6, the predicted entities varies a lot. Why?
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: see above
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper argues for importance of multimodal analogical reasoning, introduces the new task of multimodal analogical reasoning over a knowledge graph. In particular, they construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. This paper is asking an important question about the role of multimodality in analogical reasoning and goes all the way to coming up with a task to answer that question which requires constructing a dataset. While the construction might seem as combining existing datasets, the problem framing and coming up with a way to build such a task is interesting and useful for the community. Therefore, I recommend accepting the paper.
justification_for_why_not_higher_score: Because as pointed by reviewers Hqif and R5x6, the existing dataset is created in a rather simple way which has some issues.
justification_for_why_not_lower_score: Because this paper is the first to create a dataset for Multimodal Analogical Reasoning.
Decision: Accept: poster