Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work finds a simple model (called STSN) consisting of slot attention and transformer modules. Without incorporating inductive biases specific to the RPM problem format, this model still displays near-SOTA performance on the I-RAVEN and PGM datasets. Such performances suggest the importance of object-centric processing within visual reasoning.
strength_and_weaknesses: Pros:
1. It is valuable to show that object-centric models using modern modules (slot attention + transformer) can perform well on the RAVEN and PGM datasets.
2. It is informative to have ablation studies to show the improvement of each module in Table 3.

Cons:
1. It is unclear what contributions this work presents. The slot attention and transformer already exist, and this work piles them up.
2. This work claims STSN is a general-purpose architecture but do not show its performance over other visual reasoning tasks other than RPM tasks (like CLEVR [1]).
3. The model size is large compared to SCL. It is unclear whether the larger model size is a cost for general-purpose. Can you do ablation study over the size of the model (like the size of hidden dimension or the number of layers of Transformer)?

Minors:
* Abstract: "This work has ...  ", a bit confusing which work this part refers to, your work or the recent work in last sentence? (Also in Intro: "Much of this work ...")
* Sec 3.3 cnadidate -> candidate
* Figures 4 to 6 are not fit well into the page.
* The last page of the references needs re-format.

[1] Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., & Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2901-2910).
clarity,_quality,_novelty_and_reproducibility: The model architecture is clear overall, but the concrete contributions need to be emphasized.

The novelty is low, the network architecture is a patching up of already exist models (slot attention, transformer, TCN) and does not achieve new SOTA. The importance of object-centric processing within visual reasoning is already well-known.

The results seem reproducible given the details in the text, but open-source would much increase reproducibility (or indicate the open source upon acceptance).
summary_of_the_review: The contributions are not clear. As the model does not achieve new SOTA on both the I-RAVEN and PGM datasets, if the contribution goes to a general-purpose visual reasoning model, it should show its performance in other reasoning tasks that are related to objects.

---
Update (2022/11/22): raise from 3 to 5 after reading the rebuttal, comment below.

---
Update (2022/12/04): raise from 5 to 6. I hope the authors continue improving their paper (about clarity and rigorousness) and include the points they addressed in the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of training deep networks to solve visual reasoning problems. It studies PGM and I-Ravens benchmarks, which pose challenging visual reasoning problems for humans, and explores if an end-to-end deep network can solve them.

The proposed method uses slot attention to individually encode each of the prompt images in an object-centric way (ideally separating the shapes in each prompt image) and combines them in another transformer to make the final prediction. The final training objective is reconstruction on each of the individual prompt images from the object-centric representation, and a classification objective for the task.

Overall the proposed method performs comparably to the best models on the benchmarks, while claiming to be more general. 
strength_and_weaknesses: *Strengths*

- The paper studies an interesting problem in training deep networks for challenging visual reasoning tests given to humans.
- Rather than having an overly hand-crafted solution that is not general, the proposed approach does seem to maintain good generality while still performing well on the task. 
- The paper is written clearly.
- Ablations suggest all parts of the proposed model are important.

*Weaknesses*

- I'm not very familiar with this domain, so it is a bit difficult to evaluate the experiments. For example, it does seem like another approach SCL outperforms the method on both benchmarks. However, the authors claim that their proposed architecture is more general than that of SCL. The paper should do a better job of explaining exactly what ways the baselines which outperform their approach are less general, right now the claim is not well supported. 
clarity,_quality,_novelty_and_reproducibility: Overall clarity, quality, and novelty seem strong. 
summary_of_the_review: The paper does seem to propose a novel approach for learning difficult visual reasoning tasks without too much hand-crafted structure. Results seem decent but are a bit hard to evaluate without detailed knowledge of the baselines, which perform better but the authors claim are less general. Overall I would defer to other reviewers who have more knowledge of this line of work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles RPM (Raven’s Progressive Matrices) problems where a set of context images are given and a continuation image is supposed to be selected from a given set of image choices. The images are governed by abstract rules which needs to be inferred by the method, which is claimed to parallel human reasoning capabilities. The authors propose a method which largely builds on the slot attention method of Locatello et al. On two RPM benchmarks (PGM, I-Raven), the authors show competitive performance compared to prior methods.
strength_and_weaknesses: Pros
* Paper is easy to follow
* Hyperparameter choices are discussed
* Ablations show impact of each component
* Helpful visualization of the trained slot representations

Cons
* Scope of method is limited
* Limited technical novelty as the method is largestly based on prior work
* Experimental results are mixed and hard to contextualize due to lack of details about baselines
* Technical details are not described in detail
clarity,_quality,_novelty_and_reproducibility: Slot attention is based on prior work (Locatello et al.). This makes it seem like this work is an application of an existing method to this problem. The authors need to clarify the main technical contributions of this paper. 

Technical details (section 3.2, 3.3) are only verbally described. Although I can get the gist of the method, the specific details need to be discussed with better clarity (e.g. mathematical expressions). 

The reconstruction loss is vaguely described as the slot decoder in Locatello et al. Paper needs to include these details to be self contained. 

It is hard to spot the best methods in the result tables, which makes it difficult to gauge the relative performance advantage of the proposed method. 

There needs to be a detailed discussion about the baselines providing intuition about each method being compared against.
summary_of_the_review: Raising my score to 6 post-rebuttal.

This paper builds on an existing approach designed for object centric processing and applies it to abstract reasoning problems involving structured image patterns. Although the problem and approach are interesting, the novel technical contributions of this work are unclear and the paper further has issues in the experimental setup (see detailed comments).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a framework based on Slot Attention (for object-centric representation learning) and Transformer (for reasoning) to tackle the task of RAVEN's Progression Matrices. The input to the network is composed of 8 images as the context, and 8 images as candidate answers to be filled in as the ninth image. These images are generated synthetically, composed of geometric shapes of primitive shapes, sizes, colors, etc. The encoder module encodes each image into a set of slots, each of which corresponds to an "object" in the image and uses a transformer to rank all answers based on the slot-based representations.
strength_and_weaknesses: The proposed framework is straightforward and is well described in the paper. I personally very much like the simplicity of the proposed method. Tables 1 and 2 seem to be comprehensive lists of recent works on these tasks.

Despite the fact that I like the overall idea of the paper (it is simple and improves over the baselines). However, there are still a few weaknesses in the current version of the paper.

The first question is at the conceptual level. So this paper claims that “suggesting that an inductive bias for object-centric processing is a key component of visual reasoning” in the intro. I think this is one of the most important contribution of the paper (testing object-centric inductive bias in visual reasoning). However, this inductive bias has been tested many times in similar contexts, including the ALOE paper this paper cited. Therefore, this paper has a relatively low scientific contribution.

The second comment comes with my first one. If we look at the model performances, this paper does not achieve state-of-the-art performance on the testing tasks. I understand that this is because of the simplicity of the framework / not using any problem-specific inductive biases. But this makes the contributions poorly supported. One idea is to test the proposed framework on other tasks.

Regarding the non-SOTA performance on these benchmarks, I am wondering if this is because the transformer architecture is not designed for perform "reasoning" on this task. Specifically, if you look at the SCL paper from Wu et al, a major claim they are trying to make is that RPM-style tasks require reasoning about "the meta-level relationship" between concepts: understanding "progression," no matter whether it is applied to the axis of object colors or object shapes. However, Transofrmers (and other GNNs) only have object-level parameter sharing, but not factor-level parameter sharing. I wonder if this could be the reason why it underperforms several baselines.

There are a few comments about the experimental design:

1. Have you tried to train SlotAttention alone and just fix the encoding while training your transformer?
2. What's the conclusitve message from Section 4.6? Should we always use a high lamdba?

A few writing suggestions:

1. I think the title is a bit too broad. You probably want to be a little more specific.
2. The first paragraph in the intro does not have any supportive papers.
3. Given the length of the paper, I think the authors can expand their description of the Slot Attention module a bit more. This will help make the paper more self-contained.
4. Even though your model is not the state-of-the-art performer, you should still highlight the best-performing model in the tables.
clarity,_quality,_novelty_and_reproducibility: The paper presentation is clear. The experimental designs and results are reasonable.
summary_of_the_review: While overall I very much like the simplicity of the proposed framework, there are a few weaknesses that have been listed. I think the current version of the paper is not ready for publish at ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is premised on investigating if a combination of existing generic techniques / components, namely slot attention and transformer modules, are able to do well on abstract visual reasoning, specifically the Raven's Progressive Matrics (RPM) task. This model, STSN (Slot Transformer Scoring Network), does not have any task-specific inductive biases, yet does well on two standard datasets (I-RAVEN and PGM) compared to existing methods, as well as a new proposed dataset (CLEVR-Matrices) purportedly with greater visual complexity. The key claimed contribution is the demonstration that a generic inductive bias (object-centric processing) is able to tackle RPMs well, without necessitating problem-specific inductive biases.


-- STRENGTHS --

1) SOTA performance on multiple datasets, including 2 standard ones and 1 new proposed dataset

2) Method is simple and intuitive


-- WEAKNESSES --

1) Limited technical novelty: straightforward combination of existing techniques

2) Partly unsurprising results: it's already known that slot attention should work well for the RPM type of stimuli (e.g. discrete stimuli, low background clutter, limited occlusion, etc.)

3) [Minor]  Somewhat limited scope: object-centric approach implies it's ultimately limited to visual reasoning, as opposed to more generic abstract reasoning (e.g. numbers in RPM format).




justification_for_why_not_higher_score: While I recommend acceptance ultimately, to recommend as a spotlight or oral definitely seems a big stretch, due to limited technical novelty.
justification_for_why_not_lower_score: I do think that the paper, while not strong technically, does ultimately have a contribution in the senses mentioned above. The SOTA results may well be surpassed soon enough, but the field has progressed not only from distinct jumps, but also small and continuous improvements.
Decision: Accept: poster