Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel object-centric representation called Block-Slot Representation, which, unlike the conventional slot representation, provides concept-level disentanglement within a slot, such as color, texture, and position. In comparison with the previous methods, the approach demonstrated significantly better disentanglement of object properties, including complex textured scenes. Finally, the proposed method improves the results on a number of synthetic datasets (CLEVR).

strength_and_weaknesses: **Strengths:**

(1) Overall, this paper is well written, and the technical details are easy to follow. 

(2) The main idea of representing a factor of object variation, such as color, texture, and position, is nice and appealing.

(3) The main contribution of this paper is a new method for binding block factors within a slot.

(4) The experiment results support the proposed approach, including the evaluation of the DCI framework, which is a valuable addition to the paper.


**Weaknesses:**

**Experiments.** I am mostly concerned with the fact that this approach is mainly proved to be working on synthetic datasets. Thus, I wonder how it would generalize to a real domain. I think the idea is appealing, but I am honestly not sure it will work on real-life datasets.


**Novelty.** First of all, I would like to note that the main idea of this paper maintains the high standards of the conference. However, the authors should also know that I had indeed considered this idea important and implemented it 14 months ago, when the first slot-attention paper was published. The method of disentangling the structure within a slot in order to represent a factor of four object variations, such as color, shape, texture, and position, appears to be working entirely on synthetic datasets (CLEVR-like). Due to the fact that it did not work on real domains, as well as the simple nature of the contribution of the main idea (which is, to me, way too simple in this case, although there is a compelling story behind it), I decided to abandon this research direction. Despite acknowledging the importance of this paper, we should also be aware that its contributions could not be substantial enough, especially if we look at the results on real domains.
clarity,_quality,_novelty_and_reproducibility: I wrote above my concerns.

summary_of_the_review: I wrote above my concerns. I am open to the authors' feedback and other reviewers' opinions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a modification of Slot Attention to make it learn sub-components which are independent of each other (blocks), built out of prototypes which are attended to be independent RNNs. The model is trained through reconstruction (with a transformer+dVAE decoder like recent works) and performs very well on several extensions of the CLEVR dataset, including one with textures (current techniques struggle with non-trivial textured datasets). The results are strong and the presentation is very clear.
strength_and_weaknesses: 
Overall, I really liked this paper in this current form so I have very little extra to request:


1. The Abstract and Introduction are very strong, easy to follow yet entirely clear, it was really well done. Figure 1 conveys nearly the entirety of the model in one single diagram which is fantastic.
   1. The only nitpick I would have is to indicate that there are M blocks in total, this is the only missing symbol currently. 
2. The Concept Memory bottleneck is potentially both the biggest novelty or drawback of the technique in its current form.
   1. It was surprising to me that you would directly attend into it by multiplying s_nm with Cm. Somehow I would have expected something more akin to a query/key setup with additional Linear layers. Did you try something like that?
   2. The choice of making the prototypes the direct outputs is rather strong. 
      1. I feel like this may be too constraining and brittle, for example in complex situations? 
      2. I had to find how many prototypes K you used in the Appendix. You seem to use K=64 throughout, which is rather small. This seems rather much more constraining than what is done VQ-VAE/GAN or GroupVIT-style models.
      3. How “delta-ish” were the softmax attention in general? If the softmax becomes extremely peaky, this means you considerably restrict the “space” of representation (i.e. in the limit, you would only represent 64 possible values, given your choice of K)
      4. What happens if the dictionary size is much larger? What about if you share it between groups?
      5. Did you try outputting the “distance” to the prototypes instead, to have more flexibility and expressive power?
   3. Didn’t you have to use more flexible attention mechanisms, ala Gumbel, to train these prototypes well enough? I would expect the gradients to die off rather quickly in unused prototypes.
3. The Block Binding is a very interesting choice, and it seems rather useful to me so I will probably give it a try in my own work.
   1. However, I would have expected you to try the simplest choice of “just” concatenating all the groups together per slot first? I.e. provide $S=\lbrace s_n \rbrace$, where $s_n = Concat_m(s_{n, m})$.
   2. You never compared against that, even in Section 6.2? I would never have expected “the bag of blocks” to be an appropriate representation, provided as a set to the decoder. You do have the block-slots directly, so why don’t you just keep that structure? How does Figure 6a change with that choice?
4. The results are strong, and the improvement to the metric is well executed. Figure 2 is clear and very impactful.
5. Figure 4 and the methodology for obtaining it is not clear enough.
   1. How did you “mask out” part of the images? Did you use the attention weights?
   2. The corresponding section on 6.1 at the end of Page 7 needs to be expanded to provide more details, as this is a very interesting assessment which is the only way to assess what the groups and prototypes really learn.
   3. How many prototypes end up being learnt/used? Can we get some kind of “coverage” metric for their relative importance / specialization?
6. Figure 5 is strong and very interesting to see.
   1. It might be worth flagging that the model gets the reflection right (e.g. last row, second to last column on CLEVR-Hard). Do you know if this is thanks to your model, or “simply” due to the Transformer+dVAE decoder being really good at handling these?
   2. How exactly did you select the groups? You say it was easy to identify and select them, but you do not precisely say how.
7. Even though you target more complicated versions of CLEVR, this is still “just” CLEVR, which is rather easy to get results on.
   1. Did you try targeting Kubric [1] (MOVi-C onwards, discarding the temporal video aspect) or other more complex yet still manageable datasets?


[1] https://arxiv.org/abs/2203.03570 
clarity,_quality,_novelty_and_reproducibility: * See above, very high clarity.
* See above, great quality.
* Great novelty: the group-RNNs + Concept Memory choice + disentangling metric modification + Block Binding were all novel to me.
summary_of_the_review: Overall, I found this paper extremely interesting, presenting several key novel ideas which appear simple in hindsight which is always a great sign. Results are strong, the evaluation is good with good ablations and the paper demonstrates what I wanted to see and understand.

As a practitioner of this field, this is absolutely something I will try to leverage and build upon, hence I would recommend for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes a novel structured representation of scenes with several levels of hierarchy. The representation is divided into slots which provide object compositionality, and slots have several blocks of latent dimensions which are aimed to disentangle factors of variations for each object. To obtain this representation, the encoder architecture can be seen as a generalization of the well known Slot Attention component to the Block-Slot setting. The model is trained in a standard autoencoding setting, with a decoder inspired from recent work in compositional generative modeling for scenes (SLATE, STEVE), also generalized to Block-Slots via original additions to the architecture. An experimental evaluation on different variations of the CLEVR dataset clearly demonstrates the advantages of the design choices for representation compositionality, quantified as pixel-level object separation and disentanglement. 
strength_and_weaknesses: The contribution is well scoped, and obtaining good multi layer disentanglement is a very important challenge. Results in Figure 2 and 3 evidence benefits of the proposed method with a clear gap in reported metrics compared to prior work. I however have some concerns about clarity of the presentation and some experimental choices. I believe that these concerns can be addressed during the rebuttal phase. 

clarity,_quality,_novelty_and_reproducibility: 
I mainly have concerns with the high level presentation of the contribution, mostly in the introduction:
- *“the question [...] is not a small one but actually at the crux of this challenge.”* I would encourage you to improve the phrasing of this claim
- *“is often considered to be the deep representation learning approach towards high-level cognition.”* I am not sure I understand the meaning of this sentence
- *“The state-of-the-art methods for object-centric representations are currently based on Slot Attention”* I don’t specifically disagree with this claim but I think it would be good to provide more context (on which dataset and for which metrics?) 
- *“In Slot Attention, each slot finds a local area”* … *“attended local input”* could you point me to a reference showing that the attention process of Slot Attention is local (which would imply a notion of proximity)?
- *“intra-slot disentanglement has naturally been achieved in the previous probabilistic approaches”* I agree that Slot Attention is originally trained in a deterministic setting. But if this is really the main concern, why not just add a latent space regularization to the training objective in order to encourage disentanglement? It feels like this is not the main concern here
- *“make it as generic as possible by making it a simple deterministic layer like Slot Attention”* again, is this really a concern here? This sentence is followed by *"We believe that once this is achieved, it should not be too difficult to make a probabilistic extension.”*, which seems to contradict your previous statement that the probabilistic view is overly complex
- *"considering that the single-dimension representation of a factor is difficult to use as an input to a Transformer"* could you please support this claim with references and/or arguments? This is the first time I hear about this issue
- at the beginning of section 2, I find it confusing to call your slots "block-slots". I would advise to simply state that a block-slot representation is composed of slots which are composed of blocks. The term of slot is anterior to Slot Attention (for instance in the Genesis or Iodine paper) and can refer to any kind of high level unit in your latent space, so you don't need to specialise the term

Concerning design choices, the concept memory bottleneck described at the bottom of page 3 looks very similar to an attention layer, except that none of the queries, keys and values are projected. Is this on purpose, and is there a risk of capacity problem here? Is this something common in the literature or is this a design choice that is specific to your contribution?

About your structured disentanglement metric in Section 4, there are two points I would like to discuss 
- your claim that the structured disentanglement metric of Dang-Nhu doesn't apply to block level disentanglement inside slot is incorrect. In Section 5.4, Dang-Nhu applies their metric to disentanglement between the mask block and the component block inside each slot of the Genesis architecture. I think it's fine that you developed your own concurrent metric, but I would encourage you to make an accurate description of related work. Dang-Nhu also performs aggregation of feature importance inside the block, but the main difference is that they jointly optimise the slot to object matching and the probes via an EM approach. This is a more general approach that doesn't require any kind of visual information for matching. However it leads to more instability. 
- I am concerned that your experimental study compares disentanglement of representations with different numbers of dimensions. The entropy quantity heavily depends on the number of dimensions via the number of terms in the sum and the log base, and I am not aware of any reference studying the impact of increasing the number of dimensions, therefore I am not sure of how to normalise results for comparison. Given that the qualitative evaluation in Figure 3 is consistent with the quantitative results, I'm fine with keeping your results as is, but I would at least warn the reader about this issue. 

Finally, I noticed your experimental finding  *"We also note that deterministic models like SLATE and Slot Attention have significantly more active dimensions than VAE-based IODINE due to the regularising effect of the VAE prior. This observation also correlates with lower completeness-scores of Slot Attention and SLATE relative to IODINE."*, and wanted to point out that this is consistent with the systematic evaluation of Dang-Nhu, showing that the VAE loss increases intra-slot disentanglement on a variety of architectures. 

Minor:
- the letter $K$ is used twice for the size of the concept memory and the number of ground truth factors.

summary_of_the_review: This is a novel and original contribution building on the well known Slot Attention component. The technical description is clear, but I have raised a number of concerns about the high-level presentation and phrasing of claims. I am looking forward to discussion with the authors in order to make a stronger acceptance recommendation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a block-slot object-centric representation learning scheme
to use a set of sub-slot representations, aka., block representations, to explicitly encode different properties that
are conventionally encoded with only one slot representation.
The authors show the proposed disentanglement scheme performs better and enjoys better interpretability.

This idea also seems related to the Part-and-Sum detection Transformer[1] used to perform visual relationship
detection, where the authors also propose to use part queries to explicitly
encode the different partial information that is originally encoded with one sum query.
The authors are highly encouraged to discuss whether the proposed method can be applied to
broader vision tasks based on DETR such as object detection and segmentation.

[1] Visual relationship detection using part-and-sum transformers with composite queries, CVPR2021
strength_and_weaknesses: > Strengths

✅ The presented idea is interesting and the illustration is clear.

✅ The approach section is well-organized into two subsections including (i) the details of the proposed block-slot attention and (ii) how to
perform object-centric learning with a block-slot attention scheme.


> Weaknesses

❎ According to Figure 1 and the Approach section, we can see that the proposed approach might bring significant extra computation overhead compared
to the original baseline approach. The authors should report detailed complexity comparisons such as the number of parameters and GFLOPs.

❎ The authors only verify the proposed approach on the very toy benchmarks like CLEVR-Easy, CLEVR-Hard, and CLEVR-Tex.
It would be great if the authors could extend the approach to some real-world benchmarks following very recent works such as SAVi[1] and SAVi++[2]. Besides,  the authors are encouraged to discuss more details and compare the proposed approach to GENESIS-V2[3] empirically if possible.

[1] SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models

[2] SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos

[3] GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement

❎ The emergence of semantically meaningful abstract concepts in blocks shown in Figure 4 is very impressive. The authors are encouraged to conduct detailed
ablation experiments to investigate which component is the key to ensuring the emergence of semantically meaningful abstract concepts and whether this conclusion
still holds on some real-world benchmarks such as ImageNet and COCO.
clarity,_quality,_novelty_and_reproducibility: > Clarity

Good.

> Quality

Good.

> Novelty

Normal. As discussed in the related work section, there already exist some efforts along the path of within-slot disentanglement although the proposed approach seems to generalize well and does not require explicit supervision. Similar ideas also have been investigated in the Part-and-Sum detection Transformer along the path of the DETR community.

> Reproducibility

No code is available.
summary_of_the_review: How to perform within-slot disentanglement to real-world images in an unsupervised manner is a grand challenge. This work presents some encouraging results in this direction. However, the experiments are relatively weak and the authors are encouraged to extend the proposed approach to more challenging benchmarks, which might be necessary to attract broader attention from the research community. I would like to increase the ratings if the authors could well address the above concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of object discovery from synthetic multi-object scenes. It introduces an architecture called Block-Slot Attention, which extends prior work on Slot Attention [1] by introducing a further decomposition mechanism that decomposes individual object slots into so-called blocks. The resulting representation is referred to as block-slots. The authors demonstrate that this simple inductive bias results in emergent decomposition of object properties into particular blocks, e.g. one block is responsible for color, one for shape etc. – the model is validated on synthetic multi-object scenes of varying visual complexity.

All reviewers agree that this is a strong submission: it is very well-written, well-structured and overall very clear. The method is simple and appealing. The reviewers highlighted that the experiments are strong and convincing, and also mentioned that the particular choice (and improvement of) the metric for evaluation is well-executed.

The main concerns by the reviewers revolve around the simplicity of the chosen datasets for evaluation: they are all variants of the CLEVR dataset. Although the authors did include a textured version of the dataset, CLEVRTex, that has significantly higher visual complexity, no results on naturalistic or real-world datasets are presented. It is unclear whether the proposed block-slot decomposition would generalize or show benefits on more complex real-world datasets. This, however, can be excused since the community is still struggling overall to make this class of models reliably generalize to more naturalistic or real-world data, although recent progress has been made. Other minor concerns around ablations and compute requirements were well-addressed during the rebuttal.

Overall, this is a high-quality paper of significant interest to the community for which I can clearly recommend acceptance.

[1] Locatello et al., Object-Centric Learning with Slot Attention (NeurIPS 2020)

justification_for_why_not_higher_score: The paper didn't attempt to push the limits in terms of the data complexity that this class of models can address and only evaluates on fairly simplistic synthetic multi-object scenes, while other recent works have made attempts to bridge the gap to naturalistic and real-world scenes. It is unclear how the within-slot decomposition mechanism proposed in this paper would scale to more complex data. Nonetheless, the paper is of very high quality and presents a solid and elegant contribution of significant interest to the community.
justification_for_why_not_lower_score: All reviewers agree that this is a strong paper worthy of being accepted at the conference.
Decision: Accept: poster