Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an extension of SlotAtt (NeurIPS 2020) by adopting learnable queries as slot initializations. The learning process is formulated as a bi-level optimization problem, where image reconstruction is the outer objective and soft clustering of image feature is the inner objective. In order to stabilize the training procedure, this paper detaches gradients to the recursive updates, only keeping gradients for last iteration and slot initialization queries.

The proposed method is evaluated for unsupervised object discovery task on synthetic datasets and unsupervised foreground extraction task on real datasets. It demonstrates improved performance on both tasks compared with state-of-art unsupervised object-centric models, especially the vanilla SlotAtt.

Additionally, this paper shows the potential to bind object concepts to its learned slots with experiments on zero-shot transfer learning.
strength_and_weaknesses: Strength:

1. The structure of the paper is very clear. The motivation and related research are well-explained. The technical details are concise.

2. The proposed module is simple and effective, which could serve as a plug-and-play module for many models.

3. It explicitly learns object concepts from datasets as slot initializations, which paves the way for object-centric learning in more challenging cases.


Weakness:

1. More insights on learned slot initializations are expected:

Figure 3 has shown slot contents for a given input image after iterative updating. But what’s more special in this paper is the learned slot initializations, which are shared among all images in the dataset. It is expected that the learned slot initializations are the object concepts abstracted from the dataset. You may convert the learned slot initializations as images for visualization, or perform feature space analysis such as T-SNE to provide more insights.

2. More discussions on the effectiveness of new design are needed:

Although ablation experiments in Section 5.3 have demonstrated the effectiveness of the proposed module, it is still unclear why learnable queries as slot initializations can bring these large improvements. You may include more analysis and comparison with SlotAtt especially in the experiment part. For example, SlotAtt mentions that its typical solution is to distribute the background equally over all slots. However, from the visualization of this paper, it seems all background pixels are separately assigned into a single slot. A more detailed and analytical comparison with SlotAtt are needed to provide more insights on your contribution.

3. More challenging experiments are expected:

The experiment parts demonstrate remarkable results on a set of synthetic and real datasets. But for all synthetic datasets including those in the appendix, the objects are simple-colored and mostly simple-shaped, where a color-based bias may already work very well. It is suggested to evaluate on more complex synthetic datasets such as ClevrTex [1]. 

For the real datasets, all images only have a single foreground object, and the task becomes a simple binary classification. It’s more convincing to evaluate on multi-object real images, as also discussed in a very recent paper [2]. 

[1] ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation, NeurIPS 2021.
[2] Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images, NeurIPS 2022.

4. Zero-shot transfer learning experiments:

Successful zero-shot transfer learning experiments suggest generalizable representations are learned. However, if a slot learned on dog dataset can be easily transferred to flower dataset. Does it imply the slots are not necessarily binding with object concept, at least not a specific type of object? You need to investigate what object concept is learned. Are they objects such as cat and dog, or a set of properties
that defines objects?
clarity,_quality,_novelty_and_reproducibility: This paper has a clear and complete structure and is easy to read. The implementation details and experiment settings are also very clear for reproduction. However, this paper is a bit like an extension based on Slot Attention and Implicit Slot Attention (ISA). Therefore its novelty is somewhat discounted.
summary_of_the_review: This paper is a simple but effective extension of Slot Attention. The structure and presentation are great. It demonstrates remarkable improvements on unsupervised object discovery and foreground extraction. However, the major concerns are: 1) deeper technical insights/analysis/discussion are missing, 2) more convincing experiments are needed.

Consider the simple and effective design and the excellent performance, the reviwer is happy to increase the score if the concerns can be well addressed or explained.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper draws a connection between slot attention and bi-level optimization.
As a consequence, the authors suggest to learn the initial query slots rather than sampling them as in previous work. Furthermore, the design choice of Chang 2022 w.r.t. gradient propagation is adopted and validated.
Empirical results show the potential of this method for unsupervised instance segmentation on synthetic datasets and foreground/background segmentation on real-world datasets.

strength_and_weaknesses: Strengths:

- In terms of implementation, the paper proposes two straightforward design changes on top of regular slot attention: 1) initial slots are learned rather than sampled as in Locatello 2020, 2) gradient updates are skipped in intermediate steps as Chang 2022 and are propagated to the initial slots using a straight-through estimator.

- I appreciate the thorough experiments which cover both synthetic and real datasets and different decoder designs. The proposed method achieves higher scores on all datasets considered, which is a point in its favor.

Weaknesses:

- In a side-by-side comparison with the implementation of "Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation" (Chang 2022), this paper makes only one slight change in the gradient propagation. Specifically, gradients from the last step of the clustering loop are used to update the initial slots (straight-through estimator). This is a quite minor change, which may be seen as an ablation study of Chang 2022. Apart from the empirical experiments that show a performance improvement in some benchmarks, which I discuss below, I don't see any interesting theoretical insight related to this change. Sure, there is a lengthy introduction on bi-level optimization in section 3.2, but it feels artificial and disconnected from the practical change proposed in section 3.3. If this paper studies the connection between slot attention, k-means clustering, and bi-level optimization, what theoretical insights can be drawn from this previously unexplored connection? I think this is the main question that the paper should answer rather than focusing on the empirical results.

- Since the main contribution of this work is the bi-level optimization of the initial query vectors, I expected more experiments studying the effect of the bi-level optimization on the final performance. What are the implication of propagating gradients to the initial query vectors? In particular, does the straight-through estimator introduce a bias in the gradients? How different are the learned queries from the post-clustering slots? Do the intermediate clustering steps become redundant due to the straight-through estimator, essentially turning this method in a VQ-VAE? How would other methods for learning initial queries behave, e.g. a running mean of the post-clustering slots or some actual bi-level update mechanism? I think these are interesting questions that the paper should address.

- In Locatello 2020, the original slot attention is introduced as soft unsupervised clustering method that can be initialized with a variable number of random vectors. One of the most interesting experiments was the ability to train on images containing a certain number of objects and generalizing to a larger number of objects simply by increasing the number of sampled slots. Learning the slot initialization precludes this possibility because the number of learnable queries must be set as a hyperparameter, and also requires a certain a priori domain knowledge. Discussion of this drawback is lacking in the paper, instead, empirical results are presented on synthetic data where the number of object is known and on foreground/background segmentation where two slots suffice.

- The choice of how many slots to use for each dataset is not made explicit in the main text (only in the appendix). Setting the number of slots to 2 for most foreground/background tasks is very convenient but does not convince about the generality of the method. For the birds dataset it is set to 3 for some unclear reason.

- Two points regarding the ablation studies:
  - First, ablation studies usually focus on hyperparameter choices of the proposed method, e.g. number of slots or how to optimize them. Instead the ablation studies presented in the paper are a comparison with previous methods. This comparison should arise from the main experiments, not from the ablation studies.
  - Second, I am confused by some combinations of slot initialization and optimization procedure. Why were these two combinations chosen?
    - I-QSA: in QSA the initial slots are initialized at random and then learned, but they receive no gradient due to the stop-gradient operation, which makes this combination equivalent to I-SA. Unless there are other differences, I would attribute the lower performance of I-QSA wrt I-SA to an unfortunate initialization.
    - BO-SA: in SA the slots are sampled at random so propagating gradients to them (BO) should have no effect and should be equivalent to I-SA. The results are in fact very similar.

- Section 5.4 the statement that BO-QSA has the potential to be used for learning concepts is overstated in my opinion. The only supporting evidence is that slots tend to specialize on colors for ShapeStacks in figure 3. However, this behavior has been observed in other slot-based methods too, and can be explained simply by noting that even a traditional pixel clustering algorithm would capture RGB colors. I would not draw any conclusion about the ability to learn concepts from these color-based observations. The other example given in figure 3 uses the birds dataset where the number of slots has been conveniently set to 3 for foreground/background segmentation. Since the birds dataset contains images of a single subject over a blurry textured background, it's rather obvious that the slots will specialize on the subject and the background. This is not a very convincing example of concept learning.

- Related to the point above, the conclusion also contains an overly bold statement "By further verifying our findings under zero-shot transfer learning settings, our model shows great potential as a principle approach for learning generalizable object-centric representations". Though I acknowledge the transfer learning results, I would highlight that the experiment in question is foreground/background segmentation on rather simple datasets. I would not extrapolate these observations to truly multi-object real-world datasets like COCO, LVIS, or PASCAL. 

Minor points:

- Why the connection with meta-learning? Bi-level optimization has many applications and meta-learning is just one of them. Nowhere else in the paper the connection with meta-learning is leveraged to justify or enhance the proposed method. I think the entire section 2.2 could be removed without hindering the message of the paper.

- Appendix A.1, check the paragraph names
clarity,_quality,_novelty_and_reproducibility: Quality: low. The proposed approach is a marginal modification of existing methods. Rather than focusing on reporting higher numbers on tasks such as foreground/background segmentation, I would have preferred a more thorough analysis of the theoretical implications of connecting bi-level optimization and slot-based learning. Instead, the connection feels artificial and unjustified, also considering that the optimization used in practice is not bi-level.

Clarity: medium. The paper is well written, both the textual parts and the math notation. The only disconnect point is section 3.2, which happens to be the most important point of the paper where the connection between bi-level optimization and slot attention should be made explicit.

Originality: low. Both implicit slot optimization and learnable query tokens are not new ideas. The connection with bi-level optimization would make this paper original, but I find it rather weak and poorly explored.

Reproducibility: high. Implementation details are given in the main text and in the appendix.
summary_of_the_review: On the first pass through the paper, I was initially excited to read about the connection between bi-level optimization and slot attention, also thanks to the higher reported scores on several benchmarks. However, I then realized that this connection is not well-motivated and does not add any new insight. The message of the paper could have been simply "learn the query vectors with a straight-through estimator".

The paper is well written and the authors conducted experiments on several datasets, but I think the paper does not contribute novel knowledge. Surely, the empirical numbers are higher, but the choice of experiments is not on point and does not justify acceptance. I recommend rejecting the paper, but I remain open to changing my mind upon reading the rebuttal and discussing with the other reviewers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two design choices to improve unsupervised object-centric representation learning of Slot Attention. The two design choices are to learn the initial slot embeddings instead of sampling them from a Gaussian distribution, and to skip the gradients through the iterative slot refinement by using a straight-through gradient estimator. This setup is extensively evaluated in experiments on unsupervised foreground segmentation, object-centric representation learning, and zero-shot transfer of foreground segmentation.
strength_and_weaknesses: ## Strengths

- The paper proposes two simple, but yet very significant improvements in Slot Attention considering the shown experimental results. 
- The paper presents the two design choices clearly. In terms of re-producing the architecture, it requires changing only a few lines in the original Slot Attention model. Hence, the model could easily be adapted by many works in the field and be reproduced.
- To my understanding, the design choices do not introduce any additional hyperparameters and overall stabilizes Slot Attention. This is also important for future adaptations.
- The paper conducts extensive experiments on various datasets. I highly appreciated the variety of datasets, and it gives confidence that the method does not outperform Slot Attention just 'by luck' on a few cherry-picked settings.

## Weaknesses

- The paper spends a considerable amount of space on discussing the possible motivation behind the bi-level optimization, but it seems to me that the bi-level optimization is rather just an intuition on how one could connect the proposed setup to first-order meta-learning methods. For instance, what is the precise formulation of $\mathcal{L}_{\text{cluster}}$ that the model (implicitly) tries to solve? In Section 3.2, motivations are discussed for why it might be related to k-means, but it remains again a bit ambiguous. For the space used, I would have expected a clearer connection to the bi-level optimization.
- The paper misses a bit out on discussing the potential disadvantages of its method. For instance, by learning an initialization of the slots, the authors mention that they specialize to certain concepts. This would suggest that it may not work too well in out-of-distribution datasets that differ more in the 'concepts' learned. The zero-shot study is one step towards it, but the concepts remain quite similar in terms of foreground vs background. A more challenging situation is the out-of-distribution and CAMO evaluation datasets of CLEVRTEX. Since the authors already provide training results on the CLEVRTEX dataset, I believe it would be also important to report the evaluation scores for the OOD and CAMO part, even if the results may not outperform previous baselines. 
- Additionally, since slots specialize on concepts, what happens if a test set image has a larger amount of objects of the same concept than seen in the training set, e.g. 10 red objects in ShapeStacks?
clarity,_quality,_novelty_and_reproducibility: ### General
- *Clarity*: The paper is in general easily understandable, and I see no major clarity issues besides the one mentioned in the weakness section.
- *Novelty*: Both design choices have been heavily inspired by previous work and done in some capacity. For example, the learning of slot initialization was already tested in the original Slot Attention paper, and other papers have investigated the iterative behavior of Slot Attention too. However, this paper does a good job in discussing them and provides a simple yet effective way of using them. To my knowledge, the combination of the design choices is novel.
- *Reproducibility*: While unfortunately not providing code, the method seems to be straight-forward to implement from Slot Attention. Hence, I would judge that the method should be reproducible. Nonetheless, I did not try to reproduce the method myself during the reviewing period.

### Minor points

- Table 3: it would be more consistent to add the citations of Slot Attention and SLATE to the model column, even if they had been cited before in the text.
- Appendix: The appendix seems to have swapped the caption below the tables instead of above

### Typos

- Page 2, Section 2.1 (second line): "a[n] iterative attention mechanism"
- Appendix Table 10: the word 'too' appears below DVAE
- Appendix Table 10: "Sof[t]max" is missing a "t"
- Appendix: Inconsistencies in writing "DVAE" vs "dVAE"


summary_of_the_review: The proposed design choices are simple yet effective changes to a popular object-centric representation learning method. While easily understandable, the paper could be improved by having a clearer connection to its (intuitively) bi-level optimization objective and more discussions on out-of-distribution results. Overall, I think it is a good paper that is of interest to a larger community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes two tricks to optimize the training of slot attention. First, it initializes the query with learnable embedding instead of sampling from a learnable Gaussian distribution. Second, it applies bi-level optimization to the training. In practice, the slot binding process serves as the optimization of an inner optimization thus the gradient caused by this inner loop stops flowing backward. However, the learnable queries can still be updated with a straight-through estimator. The experiments on synthetic datasets (ShapeStacks and  ObjectsRoom) and real datasets (CUB200 Birds, Stanford Dogs, Stanford Cars, and Caltech Flowers) show that the proposed method achieves competitive performance. Moreover, the ablation study dissects the two tricks and proves both are indispensable. 
strength_and_weaknesses: Strength:
+ The idea is neat and well-presented. Especially, table 5 clearly demonstrates the effectiveness of the method.
+ The two tricks complement each other, which strengthens the contribution of the paper. For instance, bi-level optimization makes the gradient backpropagate to slot initialization queries. 
+ The method considerably boosts the performance on almost all the benchmarks. 


Weakness:
+ The theoretical analysis is not sufficient. Despite the superior experimental results, the theoretical explanation is somewhat missing.
+ Learnable query is not novel. Previous work [1] has adopted this variant to stabilize the training.



[1] Self-supervised Video Object Segmentation by Motion Grouping. ICCV 2021.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and easy to follow. The implementation details are specific and the pseudo-code is tabulated in the paper, which benefits the reproducibility.
summary_of_the_review: In short, it is a good paper that takes simple tricks to boost the slot attention module. One possible improvement would be the theoretical explanation. Thus, I recommend accepting the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes the use of learned slot initialization, as well as bi-level optimisation to improve object-centric learning. Experimental results are strong, with extensive experiments in synthetic and read data. All reviewers acknowledge the empirical findings and their importance. Also, all reviewers think that the theoretical contributions are not that clear, and the paper overclaims.

This paper was certainly a borderline case, details  and suggestions in the next question.
justification_for_why_not_higher_score: No clear theoretical motivation, despite the very interesting good results.
justification_for_why_not_lower_score: The results are quite positive and all reviewers agreed on that, even the negative one.
Decision: Accept: poster