Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Motivated by the structure mapping theory, this paper proposes an ad-hoc solution to address the classic RAVEN problem. The authors claim the proposed method is more general than previous methods while achieving good performance.
main_review: - The authors clearly know a lot about the literature about analogical reasoning and RAVEN. This paper provides an excellent review of related work in a concise manner.

- Technical details are largely missing. There's even not a single equation in the entire paper. This makes it difficult to believe that the proposed method would work as the authors claimed. And if it does, it poses additional questions on how general it is to other analogical problems in general.

- Results are weak. It's hard to see if the proposed method is indeed better than prior work in terms of performance. For instance, under the extrapolation setting with mixed data, the WReN is much better than the proposed one. This should be the best evidence to see if the proposed method is better.

- Figures are very difficult to understand. For instance, there's no explanation of what "filter" means in Figure 1. Bar plots in Figure 6 should have standard diveration or some other metrics in addition to the bar.
summary_of_the_review: Good literature review and motivation, but difficult to tease out the technical contributions with weak and ambiguous results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of analogical reasoning. In particular, it presents a framework for learning the Raven Progressive Matrices (RPM) task, an abstract analogy task.

In the RPM task, a sequence of three images from a source domain are given. There is some relationship that holds for the sequence, e.g. the third image is the union of the first two. Then, given an incomplete sequence of two images from the target domain, the third image must be chosen from a list of four possible candidates.

The proposed Neural Structure Mapping (NPM) system consists of two pieces. The first piece is the Visual Relationship Encoder. Given the source sequence of images, the encoder predicts the type of relationship exhibited in the sequence. This information is passed to the second piece, the Analogy Inference Engine. The architecture of the engine is assembled dynamically, according to the predicted relationship. The assembled network takes the target sequence and the candidate matrices as input and selects the completion of the sequence from among the candidates.

The encoder is trained with the ground truth relationship labels. The engine is trained using the ground truth candidate labels. 

The paper presents an experiment to test systematic generalization, in which particular attributes are held out during train time. The NSM system is found to achieve better performance.

In contrast to Hill 2019, which presents the model with semantically-contrasting alternative candidates at train time, NSM achieves good performance even when the alternative candidates are not necessarily semantically related. 

main_review: # Strengths and weaknesses
- strengths:
  - Clear explanation of motivation
  - The ability to reason with analogies seems like an important step towards intelligent systems that reason in the same way that humans do
  - The proposed method performs well against existing work and the presented baselines
- weaknesses (see detailed comments below): 
  - The information available at training time may have made the task too easy, so that meaningful comparison with existing work cannot be made
  - The setup for the task seems very simple, and it doesn't seem appropriate to call it "analogy learning" 
  - Claims about the effectiveness of NSM's components could be better supported by ablation studies
  - NSM achieves good performance when semantically-contrasting alternatives are not available at train time. But given the simplicity of the environment, it's unclear when collecting these alternatives would be an obstacle. 

# General comments:
- Sec 4: The task as it is presented in this work seems strictly easier than the task attempted by Hill 2019 or any of the baselines.  Analogical reasoning is a hard problem, and part of that difficulty is learning to divide the world into relationship categories. In the NSM approach, this information is provided in a fully-supervised manner at train time. Building this into the method seems to make the problem much easier. Thus, a fair comparison with the baselines cannot really be made, since none of the other baselines have access to the ground truth relationship at train time.
- Sec 4: Additionally, this approach does not scale well with increasing relationship types. The number of relationships is fixed, and if you want to expand the set of relationships that NSM can handle, you must manually categorize examples according to these new relationships and manually create new neural-module layouts.
- In Table 2, it is noted that lower inference engine performance correlates with lower encoder performance. It is claimed that this shows that the encoder is an essential part of the system. However, this is not necessarily the case. It could be that there is some quality that makes the encoder fail and also makes the inference engine fail. Perhaps this corresponds with a type of relationship. For example, the encoder and inference engine could both have a hard time on relationship $X$ and an easy time on everything else. One way to check for this, would be to stratify results by relationship type. The other way to directly test this would be to run the inference engine with the correct relationship and the inference engine with the incorrect relationship and compare inference engine performance. 
- I'm not entirely convinced that the system as described can be said to have "learning analogies". The domain seems very simple and far away from the type of reasoning that humans do, in which much more sensory data must be abstracted away. However, I'm aware that it is common to talk about RPM in the context of analogy learning, so I willing to concede that others may disagree. 

# Clarifying questions/suggestions
- Sec 3: Is there a reason why the "extrapolation" category is mentioned here, but doesn't appear in the results in section 5?
- Sec 4: "Instead of depending on explicitly labeled candidates for mapping relational structure,"
  - Can you explain what this means? Is it very difficult to collect semantically plausible candidates? In this domain, can't the examples be generated automatically? And doesn't the proposed NSM approach also requires access to the labeled relationship for each training example anyways? 
- Sec 4.1: Why is the encoder also predicting the object and the attribute? Only the predicted relationship matters for the Analogy Inference Engine, so why is multi-task learning necessary? If it is supposed to improve the prediction of the relationship, could you include an ablation study that shows this?
- Sec 4.2: Can you explain why the architecture for the Unary module and Binary module need to be different? A priori, either architecture seems like a reasonable choice. For example, I could imagine that for a unary relation, selecting the candidate involves knowing the relationship and features of the first two matrices and nothing else.
  - In Appendix A, it seems like both layouts perform equally well across all types of relationships. So why is there a need to dynamically choose between them?
- Sec 5.1 and 5.1.1 "contrasting, normal, as well as mixed candidates"
  - I think I'm missing something here. If I understand correctly, the candidate types have no bearing on the relationship encoder training? The candidates are never fed as input to the encoder, correct? So why are the results given according to candidate selection type?
- 5.2 "The candidate probabilities of the full context NSM model are calculated as a 1:1 ensemble between the probabilities from both the inference engine and the encoder"
  - This seems like an important piece of information that should probably appear earlier -- perhaps in section 4?
  - I'm not understanding how the encoder is used to select the candidates. The Visual Relationship Encoder as shown in figure 2 has no way to take the candidate as input, correct?
- Sec 5 Table 1: What numbers are being reported here? The title suggests that each cell contains the test perfomance on the contrasting data and the test performance on the normal data, separated by a slash. But the caption suggests that each cell contains the train/test performance.
- Sec 5 Table 1: "Our approach achieves the best generalization accuracy for the most number of possible train/test candidate scenarios across all models (4/12)"
  - Where does the "12" come from? Should it say 36, since there are 36 cells in the table?
- Sec 5: Do Table 1 and Figure 6 show the same information? If so, it makes sense to only keep one of them.

# small suggestions
- Figure 5: The y-axis can be scaled down
- The numbers on Figures 5 and 6 were a little too small
- If Table 1 and Figure 6 show the same information, one of the them can probably be omitted.
- pg 6. "fed parallely" --> "fed in parallel" perhaps?


summary_of_the_review: The problem of learning to reason with analogies is important, and this paper makes an attempt towards doing this in a structured way, which seems like a principled move. However, as it is currently, I cannot recommend the paper for publication. The task being learned has been made too easy, so that claims about effectiveness cannot be supported. But if the authors work on a version of this study, in which the relationship types are latent/unobserved at train time, I think it would be a very promising step towards better analogical-reasoning systems.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new architecture for learning visual analogies, based on Gentner’s Structure Mapping Theory for how humans might draw analogies. Gentner’s theory proposes representing the relationships between objects explicitly, so that this relational structure can be reused in new domains (and suggests that this commonality in structure is what permits analogies to be made between perceptually dissimilar objects). The authors propose a neural network model architecture and test it on the Raven’s Progressive Matrices dataset. The proposed architecture first splits a series of ‘source’ visual scenes into objects, attributes and the relationships between those scenes, before feeding just the relationship head into a second network. The second network then switches between two different architectures (depending on the relation fed in). The architecture in the second network (whichever is chosen) receives the ‘source’ relationship and two ‘target’ scenes before trying to predict which of a set of 4 candidate  ‘target’ scenes completes the visual analogy between source and target. The authors test their architecture on the generalisation splits in the RPM dataset and compare test accuracy results to the baseline models used by Hill et al, 2019. The authors show that their model (which builds in additional architectural structure) performs better at a subset of tests than more general architectures.
main_review: Strengths:

Interesting proposal to implement an architecture that embodies a classic theory of analogy learning (Gentner 1983). 
Visual relationship encoder architecture was clear and logical, and the relationship to Gentner’s Structure Mapping Theory was clear in this component, as the Visual Relationship Encoder outputs an explicit representation of ‘structure’, which is here manifest in the relationship between separate visual scenes. 


Weaknesses:

Clarity of text, theory and justification for work:
The paper was difficult to read in parts, as some concepts were poorly explained. For example, the main paper that this work is based on seems to be somewhat misrepresented at several points in the text.  “It is not feasible to build or curate datasets to always exploit the structure mapping prior”, “their key contribution was to introduce the SMT prior into the dataset…”. “Hill et al (2019) hypothesized that introducing a prior on the learning process that requires the learning to correctly identify the relationship would align with how humans learning structure for analogical mapping”. The work by Hill et al demonstrates that when neural networks are trained with candidates that embody contrasting relational structures (akin to a curriculum), this enables the networks to learn to make analogies more effectively.  It is not clear what the authors mean by “Structure Mapping Theory prior” in these and other places in the text, but it does not give a good intuition for what this highly related piece of work contributes to the literature. When relating the current paper to other work, the authors also do not explain specifically how their architecture differs from those proposed in previous work, which is important as the architecture is the point of novelty in this paper.

The authors justify this work by making the point that in order to learn to make analogies, a researcher cannot rely solely on using class contrasts when presenting candidate options (i.e. the method proposed by Hill et al 2019), as these require knowledge of which relationship each candidate embodies, knowledge which may not always be available to a labeller. This is a fair point to make, however here the authors are proposing a new architecture that they also train and test using the contrast method illustrated by Hill et al, so it is not as though this paper proposes an alternative to class contrasts in training. In fact, the results show that the authors’ method works best in comparison to other methods precisely under the Hill et al contrast conditions. As far as I can see, there are no test splits in which the architecture proposed here performs best under Normal (not contrast) training and test conditions. The argument that clever contrast training is “not feasible” therefore seems a little misplaced.

Interestingly, in the appendix the authors show that the adaptivity of their inference engine does not enable better analogy inference than either of the static engines. It makes the reader question why the authors left this extra complexity in their design when they show it has no impact on performance, and presumably requires additional resources to train (many more weights for this extra section in the Inference Engine). The authors do not provide a clear logical reason for its inclusion. 

Results:
The generalisation results appear to be somewhat cherry picked. The previously published dataset the authors use includes 5 generalisation splits, which each assess a different component of systematic generalisation. However the authors present results from just 2 of these 5 splits in the main text (Table 1 and Figure 6), and no explanation is given for why these 2 splits were the ones presented. In appendix D the authors do provide results for their method across all generalisation splits, but there is no comparison to other methods here so the values are hard to interpret. To properly and fairly communicate their results, the authors should present data (as in Table 1) for all generalisation splits and compare these fully to other work.

Figure 5 presents performance results for just the Visual Relationship Encoder, but split across all 5 different generalisation splits and also across different contrast conditions, as though each of these combinations of conditions tests a different hypothesis. However, as the encoder takes as input only the first three (source) context squares, and generalisation splits are constructed so as to isolate different relationships between the source and target squares, these many (45) bars dont tell the reader much more than what a single bar would have communicated: that the Visual Relationship Encoder predicts with about ~85% accuracy. Correspondingly the bars are all virtually identical. Presenting them all in this way implies that the Visual Relationship Encoder might be somehow performing different types of systematic generalisation, when this is really not the case.

Finally, it is nice that the authors try to determine how important the relationship output from the Visual Relationship Encoder was for performance of the Inference Engine. However the way this specific experiment was run confounds the usefulness for the Inference Engine of knowing the relation per se, with potential correlations in performance between the two network modules on each analogy example. The correct way to run this experiment would be to artificially provide the Inference Engine with all possible relationship labels (a different 'fake' relation label per run), and then evaluate whether the Inference Engine performed better when the relationship provided matched the true underlying relationship.

summary_of_the_review: Overall,  this paper tested an interesting hypothesis based on a classic idea from cognitive science. However the results were not thoroughly analysed, and there was a  lack of clarity in the explanation of ideas that made it hard to situate this contribution of this paper in the literature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper targets the problem of abstract reasoning, with a special focus on the task of learning visual analogies. The authors propose a multi-stage neural network (Neural Structure Mapping, NSM) for decomposing the problem into vision relationship recognition and concept inference. They tested their model on an existing RPM (Raven's Progressive Matrices) based visual analogy benchmark that contains different systematic generalization tests and outperformed existing models. The authors made further discussion on these experimental results to support their proposals on model designs.
main_review: This paper focused on learning abstract visual analogies, which is an important problem for machine learning models to be able to generalize. To illustrate the effectiveness of learned analogies, the authors chose a carefully designed benchmark that aims at solving generalization problems under the domain of RPM problems. This benchmark is deliberately designed to challenge models' performance on the task of novel domain (novel concept), novel domain type (novel concept composition) and novel domain values (unseen values for known concepts), which from my perspective, is sufficiently difficult as a starting-step task and a good direction to go. However, I strongly feel that the authors did not distinguish their work from existing ones. Given that the benchmark, generalization tests, and learning setup (learning by contrasting) are defined in prior works, the major contribution of this paper should have come from model design and learning method, but the novelty of the proposed model is somewhat weak. First of all, the idea of leveraging object-centric representations for solving abstract reasoning tasks has been widely discussed in RPM solving tasks, which makes the visual relationship encoder conceptually not a new thing. Further, the proposed model claimed to have borrowed ideas from modular approaches for the analogy inference engine, but the unary/binary operators mentioned are designed specifically for this RPM-based task and have fixed structures (though the authors discussed dynamic structures in the appendix). This makes the proposed two structural layouts a bit weak as (1) it lacks semantic information when compared to modular networks (does not induce anything we could use for other similar tasks), and (2) it might not meet the need for more complex scenarios (e.g. real-world analogies) as a structural mapping engine. This is somewhat supported by the fact that ResNet-Parallel could be achieving similar results under certain training settings. Given these facts, the proposed model seems to be constrained on the current task and does not have strong potential as a general solution to visual analogy learning. 

Here I list some points that the authors could clarify or correct me if understood them wrong during rebuttal. 
(1) To find the best model that can generalize well given the current test settings, I assumed we should be looking at normal testing accuracy on each training regime since compared with contrasting candidates, the randomly sampled candidates can be more confusing. If this is the case, should we be looking for the best model that could achieve the best normal testing accuracy? (in this case, perhaps ResNet-Parallel or WReN?)
(2) It is a bit hard to see the difference between training setups and target tasks except for the obvious gap of novel domain transfer. As all visual encoders are trained given only the source domain context panels, one could imagine this task should not be too different for each generalization split. Then is there a potential reason why such a gap exists for different training setups or splits? Is it because of the joint training of later modules or distribution shift in the benchmark itself?
(3) There are minor errors on paper writing including typos and figure errors (e.g., the highlighting candidate in Figure.1 is incorrect).
summary_of_the_review: Although this paper works on an interesting and potentially impactful problem, the current design of the model and illustration does not fully support the claim the authors made and is somewhat incremental on technical novelty. Therefore, I recommend a rejection at this time with the hope that the authors can make this submission stronger in the next version.

Post rebuttal:
After reading the authors' response, I still feel that the current submission is somewhat incremental and needs more solid results for justifying the authors' claims. Scores are increased from 3 to 5 given the clarifications of the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A model called Neural Structure Mapping (NSM) is introduced to solve the task of abstract visual analogy making. The NSM model consists of a visual relationship encoder and an analogy inference engine. The visual relationship encoder extracts the visual domain elements, including object, attribute, and relation, while the analogy inference engine is a neural modular architecture that constructs the model layout based on the relation and predicts the final answer. On the dataset proposed by Hill et. al., the NSM shows better performance than other baselines.
main_review: I like how the authors motivate the work in a way that can be connected to the classic structure mapping engine, as this theory is an agreed-on approach in cognition for how analogy is made. And certainly this theory could be better leveraged for the learning community to solve the task of interest in this work. I also appreciate the engineering efforts the authors make in this work that notably improve over earlier methods in this challenging task.

However, I'm particularly concerned on the novelty of this work. Using a neural modular architecture is anything but new in the reasoning community. Earlier works have extensively explored the potential of such an approach in VQA, and moreover, the NS-CL method further manages to jointly train both the visual component and the reasoning component. In this work, however, the visual component is neither jointly trained with the analogy part, nor is the analogy module any different from existing modular construction methods. What's worse, there are only two possible network layouts, a much simpler design than existing cases in VQA. Therefore, I only see this work as an application of the existing modular method in a new domain, with very little novelty. In addition, the DRT model in Zhang et. al. is exactly the modular approach, despite the fact that they call it DRT instead of neural modular network. And that model was among the very first trials in such tasks.

Besides, the authors motivate this intuitive method with SMT, which becomes no more than a story wrapper in the paper. The theory is there, but apart from *claiming* the theory and the model are related, there is not any connection between the theory and the model: in what sense is analogy inferred? The method just simply fixes the structure and selects it based on a predicted label trained from ground truth.

For the experimental evaluation, I'm not sure why the authors only select the simple models to compare. Over the years, there have been quite some new works in this topic, some have already been mentioned in the literature review, *e.g.*, MXGNet, LEN, Rel-AIR, SCL, et. al.. What about comparing with them? I also notice that the dataset used is an incomplete version the traditional Raven task. What is the model's performance on the full matrix?

Besides, if you only need the relation, why bother predicting object and attribute?
summary_of_the_review: I'm concerned on the novelty of the work and the evaluation performed.

I have read the rebuttal and other reviews. As the authors agree on most of the comments, I decide not to raise the score. But I do acknowledge that
1. The DRT may not fall into the modular network domain though the idea looks very similar.   
2. I cannot quite connect the model and the structural mapping theory despite other reviewers' opinion.
3. I would still encourage comparison with other models as the tasks considered are related.
review_title: null

Meta-review: This manuscript presents an approach to handling abstract visual analogy tasks where panels of drawings are shown with a missing entry. One of a number of candidate drawings must be chosen to complete the panel. Reviewers brought up several concerns:

1. The task performed was made considerably easier by providing additional annotations at training time. This was not the case in the original task in prior work that the manuscript builds on. No convincing explanation was provided as to why this change is critical to accommodate the manuscript's contributions.

2. A key feature of the approach, the adaptive modular design, does not seem to contribute much. The authors rightly point out this may be a limitation of current benchmarks. Reviewers were sympathetic to this view but that leaves the manuscript in a tough spot: a central contribution cannot be evaluated. What is even worrisome is that without evaluating the effectiveness of the adaptive design we cannot know if it is working at all. What if adaptivity is required for some future analogy tasks but it turns out that this approach, despite seeming to be adaptive, falls short?

3. Another contribution, the multi-task encoder does not seem to provide much value in ablation experiments. The manuscript would be improved if this feature was removed or its usefulness was demonstrated.

A number of smaller issues were also brought up by reviewers.

Throughout the responses to reviewers the authors highlight that their central contribution is incorporating a structure mapping prior "The central contribution of our method is introducing a structure mapping prior ...". I would like to draw the author's attention to the fact that they had to remind 3 of the 4 reviewers to focus on this rather than another aspect of the work. That clearly indicates that the manuscript and work needs a shift in focus. I would suggest that authors double down on their structural mapping prior, eliminate all other features which turned out to be controversial or impossible to evaluate, and demonstrate the utility of their idea in two domains, i.e., including another domain. This would really highlight the core contribution.

Unfortunately, what may turn out to be a good idea, the structural mapping prior, is lost among many other complexities. I hope the authors are not discouraged and that we see this line of work again in the future.
Decision: Reject