Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper proposes a neural network based approach called Dual-Contrast Network (DCNet) to solve Raven’s Progressive Matrices (RPM). The approach consists of a rule contrast module that compares the latent rules between the unfilled (third) row/column and the filled (first and second) rows/columns, a choice contrast module that helps in picking the correct choice among the given eight choices, and finally uses a 2-layer MLP to predict scores for the choices. Different from previous approaches, the only supervision used in the proposed approach is the ground-truth choice. The approach achieves state-of-the-art performance on RAVEN and PGM datasets.

——————————————————————————————————————————————————————————————


Strengths:

S1) The approach achieves state-of-the-art performance.

S2) The approach requires the least amount of supervision — the ground-truth option, unlike previous works that require auxiliary annotations or assumptions.

S3) The paper shows the contribution of the approach’s two main modules using ablation studies.

S4) The approach outperforms previous works when trained with smaller training datasets. 

——————————————————————————————————————————————————————————————


Weaknesses:

W1) The results lack standard deviation or error bars in the tables. Are the differences from previous works statistically significant?

W2) I think the paper would benefit a lot with further analysis into the results — what type of mistakes do previous methods tend to make which the proposed approach is able to overcome? The paper should contain qualitative examples showing what options previous methods predict, especially when they fail and the option predicted by the proposed approach. 

W3) Similarly, the paper should analyze the failure modes of the proposed approach. Is the task solved? If not, what obstacles still remain?

W4) How does the computation complexity of the proposed approach compare with previous works?  

W5) The paper claims that the ability to solve RPM correlates well with human intelligence. But the few examples shown in the paper seem super hard. How well do humans perform on these datasets? I believe it should be much lower than that of the previous works as well as the proposed approach. If that’s the case, why do we want to build better and better approaches for this task?


——————————————————————————————————————————————————————————————
——————————————————————————————————————————————————————————————

Update after rebuttal: I thank the authors for their responses to my questions. They satisfactorily answer most of my concerns. Overall, I agree with the concern that the proposed approach is specific to RPM and it's unclear how well it (or parts of it) would generalize to other problems but I think the approach is quite interesting, novel and achieves state-of-the-art results. Hence, I think the contributions of the paper are significant for acceptance. 
review_title: Interesting approach, impressive results

final_score: rating: 7: Good paper, accept
review_content: 
Summary
---
This paper proposes a new approach that improves performance on the Raven's
Progressive Matrices problem without auxiliary annotation by using two
complementary principles to structure inference.

(introduction)
The RPM problem is a visual reasoning problem where a 3x3 grid of images is provided
with one empty slot and 8 choices to fill in to that slot. The goal is to chose
the correct image via reference to the logical structure of the grid.
Two structures stand out:
1) The same rules apply to every row/column.
2) Many of the incorrect choices obey some, but not all of the rules.

(approach)
This paper incorporates both structures into its neural net design.
(structure 1) Each of the 2 provided rows and the 8 possible completions of the 3rd row
are embedded into a common space, and each choice is represented as a difference
between that choice's embedding and the embedding of the 2 known rows.
(structure 2) Next the approach considers the average embedding of the choices
and represents each choice by its position relative to that centroid.
This representation is fed through an MLP-sigmoid configuation to get a score
for each choice and perform the final classification.

(experiments)
Experiments compare to many recent state of the art baselines.
1. The proposed DCNet outperforms all baselines, without or without auxiliary supervision, on the RAVEN dataset.
2. The proposed DCNet outperforms all baselines, except those with auxiliary supervision, on the PGM dataset.
3. Performance degrades when the amount of training data is decreased, but not as much as CoPINet.
4. Ablations show that both of the structures incorporated into DCNet help improve performance.



Strengths
---

The RPM problem is interesting.

The dual contrast approach is well motivated.

The proposed DCNet outperforms relevant baselines on both datasets.

Ablations help understand the importance of the two types of structure and how performance improvements are distributed over subsets of the dataset.



Weaknesses
---

All the weaknesses I found in this paper were pretty minor. A few parts weren't as clear as I would have liked and the paper probably won't have much significance outside of the literature on RPM.


Clarity:

* The notation in the 2nd paragraph of 3.1 is a bit confusing. A figure or more specific example might help clarify. Is r_i,j a single image or all three images from a row concatenated together? Is each input image a different channel or are they concatenated?

* Multiple times the paper mentions that a disadvantage of the CoPINet approach is that it requires knowing the maximum number of possible rules and attributes. It would be nice if the paper elaborated on what kind of conern this is. Is this bad because it's not satisfying? (I agree that it isn't.) Or is this bad because of some practical issue? (Does it have a real effect on performance in the considered datasets?)

* Is one of the two contrasts generally enough to solve the problem in theory? Do these two contrasts just make inference easier, or do both really need to be considered by any system that solves the problem in general?


Significance:

* The correlation of RPM scores with general intelligence in humans makes the problem interesting on its own; more than a toy problem. But the approach is still very specific. I doubt very much from this approach will generalize to other problems.

* I understand what makes the approach different, but I don't have a qualitative assessment of how performance changed. What sorts of problems do other models fail at that this DCNet succeeds at? Maybe there's not an intuitive answer to that question which has been found in investigations so far. If so, I'd find it useful for the paper to say that.

* The performance improvements aren't very large, though I think they are probably significant, and that this approach would be a useful contribution even if it didn't beat all the baselines. Still, it would be useful to get a sense for the variance over random initializations and bootstrapping (i.e., confidence intervals).


Other:

* The choice of sigmoid instead of softmax in section 3.4 is a bit odd. Exactly one of the choices is correct, so it seems more natural to normalize them together instead of independently.


Preliminary Evaluation
---

Clarity - The paper is pretty clear overall, though some parts could be improved.
Quality - The approach is well motivated and the experiments include lots of ablations and comparisons to baselines that make the conclusion well supported.
Significance - This will likely have some small significance in the RPM literature. Improvements are small and the approach is fairly specific to RPM. However, the improvements are meaningul and the approach is interesting.
Originality - Previous approaches to RPM problems have taken advantage of the problem structure, but this specific structure in this way.

All of the factors above clearly point to acceptance, though general interest in this paper will be limited.



Suggestions
---

* The related work specifies that CoPINet also takes advantage of contrast. Since contrast is the central motivation of this paper, I wanted more detail about how exactly they do that. It would be nice to expand the related work to provide a bit more detail about contrast in CoPINet.

review_title: Interesting new approach to RPM

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper proposes a new approach for abstract reasoning and explores it in the context of the RPM task. In contrast to other competing approaches, the authors seek to build into the model as few assumptions as possible to keep the model general and not specific to the specific problem or to particular annotations or supervision signals. The general capability that they seek to incorporate into the model is the ability to effectively compare and contrast candidates in tasks that require choosing the best fit. 

The task is important, presented carefully and is well-motivated and the paper is clear and easy to follow. The related work section covers the necessary backgrounds including both visual reasoning and general and the RPM task in particular. It also presents the existing methods and discuss their disadvantages compared to the new approach - mainly their stronger reliance on supervision to learn good, semantic or disentangled features that will help most in addressing the task, or particular assumptions other approaches make about the specific properties or structure of the RPM task that may not hold in others or more general cases. 

The task is also clearly presented and the authors explain how it consists of two main sub-tasks: (1) identifying the rule that links the existing rows or columns, and (2) comparing the candidate answers to choose the best fit. They propose network modules to solve each of these tasks correspondingly: a rule-contrast module, and a choice-contrast module, to complete each of these sub-tasks. For both  sub-stack they use clustering to find contrast and similarities between elements. This idea is simple, nice and quite novel I believe in these contexts. 

Experiments are performed over two datasets of RAVEN and PGM, and achieve 5.77% improvement on average over state-of-the-art, and larger improvement in scarcer-data regimes, which the authors particularly focus on. They provide useful information about the datasets, baselines and implementation details and experiment settings, along with an ablation study to give further insight into the benefits of particular aspects of the model.

A particular comment that I have is that it seems that the model considers each answer by replicating the board k times for the k candidates, and considering each such completion alternative. While working for the RPM case, it won’t scale to problems where there’s a larger number of candidates, potentially even not-bounded. Restructuring the network to be able to preprocess as much as it can about the board so to reduce the amount of answer-dependent computation may be very useful to make the approach more general and efficient. I’m also looking forward to hearing about further applications of this idea as discussed at the end of the paper in the general VQA task or on other abstract reasoning, for instance such as the Abstraction and Reasoning Challenge.

Overall, Great work!

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed an inductive-bias contrast module that improves performance on Raven Progressive Matrices datasets. The proposed module makes use of the prior knowledge that rules can only exist in rows and columns of the diagram matrix. 

While the paper claims to out perform previous SOTA, I have to point out that the authors are missing citations of the more recent SOTA methods on PGM and RAVEN datasets, such as [1] and [2]. If you take into account these more recent results, the proposed model only achieves slightly better results on RAVEN dataset, while not achieving SOTA results on PGM dataset.

There is some (but limited) novelty in using the prior knowledge of RPM rules to design inductive-bias module that works for this specific tasks. However the module is tailor designed for RPM tasks, which means it cannot be easily adapted to other types of reasoning tasks. Thus I feel the paper will only have impact within the RPM community, but little impact in the general ML and Reasoning community.

While the authors reported scores on PGM and RAVEN datasets, the authors did not report generalisation performances on various generalisation splits of PGM and RAVEN datasets. These missing results are in fact more interesting, as PGM and RAVEN datasets are designed for measuring generalisation capability of neural models.

In summary, I vote for reject as this paper is not citing the more recent SOTA results, is tailor-designed only for RPM tasks, and lacks experiments on the generalisation split of PGM and RAVEN datasets.

Clarity:
The paper is clearly written and easily understandable

-----------After Rebuttal--------------
I decide on up my score by 1 since indeed the authors are only required to cite papers that are peer-reviewed and published before 2nd August. I decide not to up my score any further because my other two concerns remain. I appreciate that the authors performed additional experiments. While the generalization performance is better than MLRN, it does not outperform other baselines such as MXGNet. 


Reference:
1. Kim, Youngsung, et al. "Few-shot Visual Reasoning with Meta-analogical Contrastive Learning." NeurIPS 2020 (2020).
2. Wu, Yuhuai, et al. "The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning." arXiv preprint arXiv:2007.04212 (2020).
review_title: Paper 576 Review

Meta-review: This work proposes a new architecture for solving Ravens Progressive Matrices (RPM), a well known form of visual reasoning problem. The method relies on an operation that directly compares the final row and column (completed with different candidate answers) with the first two rows and columns. Doing this allows the network to perform better than previous approaches when measured on an in-distribution test set on two RPM datasets, RAVEN and PGM. 

As the reviewers pointed out, a strength of this work is the strong performance on the neutral split of these datasets and the fact that the methods do not (unlike some other approaches) require access to any annotations from the dataset other that knowledge of the structure of the RPM task and access to the candidate answers and the correct answer. 

However, a noted weakness is the fact that the network reflects the structure of the task more directly than other approaches, which means that the insight is specific to the problem of solving RPMs. Another weakness is that the authors focus on the neutral (in-distribution) splits. Reading the PGM paper, it is clear that the neutral split is not really the main focus of that dataset (it accounts for only 1/7 of the dataset), which seems to have been specifically developed as a benchmark for out-of-distribution generalisation. Indeed, the whole point of the RPM task is to measure the ability to induce abstract rules and principles from pixels, but without measuring out-of-distribution generalisation, can we really claim that any model has induced a 'rule'? 

The authors mitigate this issue to a small degree during the rebuttal by adding scores on the 'interpolation' and 'extrapolation' splits of the PGM dataset, but still do not consider the other splits where rule application is most clearly tested. 

I note that the weakness described above also applies to lots of other published work involving PGM and RAVEN datasets. 

In summary, this is a well-executed, neat piece of work that shows a better way to fit a large dataset by incorporating knowledge of the structure of the data into the task. Because it does not consider the full benchmarks, only the in-distribution splits, it falls short of showing that this enables better induction of abstract principles or rules. On the majority opinion of the reviewers, and because there are no scientific flaws in the work, I recommend acceptance with weak confidence pending wider calibration across the program. 
Decision: Accept (Poster)