Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an object-centric representation learning model that targets large-scale real-world datasets. To do so, they propose to learn the object-centric representations by reconstructing the features of a pre-trained encoder (DINO). The hypothesis is that the features pre-trained on large-scale datasets contain semantically meaningful information which provides stronger semantical bias than the low-level features for object grouping. To evaluate the method, the paper provides quantitative and qualitative evaluations of segmentation quality on synthetic datasets with object-centric baselines and real-world datasets with unsupervised semantic segmentation baselines.
strength_and_weaknesses: Strengths:

- The topic of developing scalable object-centric representation learning models is of significance to the community.
- As the first paper to test an object-centric model on large-scale real-world datasets like COCO, it is a promising endeavor in this direction.
- The experiment results show that the proposed method achieves improvement over previous object-centric learning models (though containing questionable results) in segmentation and object detection.

Weaknesses:

- Lack of evaluation of the representation quality
    - The proposed method is only evaluated on object detection and segmentation tasks. The quality of the learned representations, which is one key aspect of representation learning, is not evaluated. Additional downstream tasks using the slot representations, such as properties prediction, should be included to address this problem.
- The quantitative results of the baseline model SLATE [1] are questionable.
    - The experiment results on the baseline model SLATE show very poor performance on object segmentation compared to other models including slot attention. In the MOVI series dataset, SLATE’s mBO score is even close to or worse than the Block Pattern baseline which uses predefined regular block masks. This is questionable, as we have also conducted a similar experiment on various synthetic and real-world datasets, and SLATE outperforms slot attention on all datasets for object segmentation and could achieve higher scores than that reported in the paper for the MOVI datasets. Some investigation might be needed to address this question.
    - The corresponding qualitative evaluations are also missing. Please include them in the qualitative evaluation.
- The claim of the method is to achieve object-centric learning in real-world images. However, the method seems to provide only semantic-level segmentation on real-world (or possibly all) datasets.
    - The experiments show that the method tends to provide semantic-level segmentation on real-world datasets while instance-level segmentation on synthetic datasets (MOVI). Since the MOVI dataset rarely contains multiple same-class objects in one image, i.e., instance-level segmentation and semantic-level segmentation might have similar results. Therefore, the model could actually be doing semantic-level grouping on all datasets. Please verify this part.
- This is important. Semantic segmentation baselines should also be included in the MOVI experiment
    - Given the fact that the proposed method does semantic-level grouping in real-world datasets, it seems necessary to include some semantic segmentation baselines in the object-centric learning tasks. More specifically, please provide the set of MOVI series evaluation tasks on some of the unsupervised semantic segmentation baselines in Table 4 with proper slots/clusters number.
    - This is important because it demonstrates if the existing unsupervised semantic segmentation models already solve the object grouping problem in the multi-object datasets. This might reveal some limitations of existing synthetic multi-object datasets commonly used in evaluating object-centric learning models. The comparison results will also be insightful in explaining the difference between the method and the unsupervised semantic segmentation baselines.

[1] Singh, G., Deng, F., & Ahn, S. (2021). Illiterate DALL-E Learns to Compose. *arXiv*. https://doi.org/10.48550/arXiv.2110.11405
clarity,_quality,_novelty_and_reproducibility: - The paper is easy to follow for the most part.
- The model design is limited in novelty. The slots encoder and autoregressive decoder are both proposed in prior work. The idea of reconstructing features instead of pixels using slot is novel.
- The paper seems to provide enough information to reproduce the results.

Additional question:

- Is there a reason why mBO is used instead of mIoU which involves object matching and could potentially provide a more accurate evaluation of the segmentation quality?
summary_of_the_review: This paper aims to develop scalable object-centric learning models for real-world scenes. The topic is important, and the proposed model is the first attempt in this direction. Experiments also seem to suggest that the method provides an improvement over the prior methods. However, as explained in the weakness section, the evaluation contains key problems to be addressed. I am willing to amend my score, if the questions are properly addressed.

---

## Post-rebuttal updates:

Since the rebuttal has partially addressed my concerns, I am now more inclined to accept the paper and have raised my **Recommendation** score to 6. However, due to some of the issues outlined in my response to the rebuttal, I still find this paper on the borderline. See [here](https://openreview.net/forum?id=b9tUk-f_aG&noteId=8vp4F8U2You) for details.

This paper presents an interesting topic, yet further investigation is needed to answer the lingering questions. The authors should ensure a more comprehensive exploration in the camera-ready version.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper brings together progress made in two parallel fields – object-centric learning and self-supervised learning. In particular, SlotAttention, one of the most popular paradigms for object-centric learning and unsupervised scene segmentation, is known to fail when the visual complexity of scenes increases, as for example in the case of real-world imagery.  The authors show for the first time that it is possible to successfully tackle real-world scenes with a SlotAttention-like architecture, by modifying the learning objective from image reconstruction to feature reconstruction (using self-supervised features from DINO). 
strength_and_weaknesses: 
**Strengths**
                                                        	
1\. Real-world data has been a notoriously difficult task for unsupervised scene decomposition methods, such as SlotAttention, IODINE, Monet, etc., that learn object representations. The paper takes a simple but important step to bridge the gap to real-world data and is the first SlotAttention-like approach to be applied to datasets such as COCO and PASCAL VOC with success.  

2\. The method is well-evaluated with numerous and informative experiments, ablations, and promising results on MOVi-C/E, PASCAL VOC 2012, COCO, COCO-Stuff. The figures that are provided are also useful to understand the quality of the predictions beyond quantitative measures, especially since the discovered classes do not necessarily align with ground truth ones.

**Weaknesses**

1\. Since the image encoder seems to play a significant role in making SlotAttention work for real-world scenes, I think it would be extremely useful if the authors could evaluate different types of self-supervised features, besides DINO. Other options include MoCo-v3 or MSN, which are all also based on the ViT architecture. This would provide a lot of insight into whether the choice of features is an important aspect to consider, which I think is the main thing currently lacking in the paper.   
  

2\. The different figures in the paper, including the one analyzing the number of slots (Fig. 6 in the Appendix) suggest that, for the most part, the model is using all possible slots to model the scene. The paper also discusses this, mentioning that a fixed number of slots is often not appropriate. However, to the best of my knowledge, this is not the case for SlotAttention on simpler datasets, where the number of slots corresponds to a maximum number of objects that can be discovered but often the model uses fewer than all available slots. Therefore, it appears that the proposed method does not fully inherit the properties of the SlotAttention mechanism, but rather appears to have an effect akin to clustering DINO features.  

3\. As discussed in the conclusion, another property of SlotAttention that the proposed model did not inherit is the ability to separate objects as different instances of the same semantic class. Interestingly though, at the task of unsupervised semantic segmentation, this method shows the weakest performance. 

Considering points 2 and 3 it would be useful to have a discussion on what actually is inherited from SlotAttention and what contradicts the goal of object-centric learning, which is to model the compositional properties of scenes and is central to methods such as SlotAttention. On a higher level, is it possible to make object-centric learning work on real scenes without stepping away from mechanisms that meet this goal?  

4\. It would be useful to provide a more extensive discussion wrt to ORL [1] and other object-level representation learning papers, such as Odin. The authors discuss Odin very briefly mentioning that they focus on semantics rather than instances. However, as I elaborated above I think this is rather a by-product or limitation, i.e. a property that this model unfortunately did not inherit from SlotAttention.  

---

[1] Xie et al., Unsupervised Object-Level Representation Learning from Scene Images, NeurIPS 2022


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and pleasant to read. Both writing and execution are of high quality.

Reproducibility: A lot of implementation details and hyperparameters are given in the Appendix. Public code release is promised upon acceptance. 

Novelty: This method is essentially the combination of DINO and SlotAttention so the novelty is somewhat limited. However, along these lines, the paper presents many interesting findings that are relevant for both object-centric and self-supervised learning fields. 

summary_of_the_review: The paper achieves a very challenging task for object-centric learning methods, which is the application to real-world data. The way to do so (self-supervised features) and the findings presented in the process are interesting and have the potential to influence future work. For this reason, I am inclined toward accepting the paper. Some limitations do exist and are acknowledged by the authors, for example the fact that the method results in a semantic rather than instance grouping of image elements, which is worth investigating further.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an object-centric learning approach that scales to real-world data. The method builds on Slot Attention, but instead of reconstructing pixels, it reconstructs image features obtained by an encoder (e.g. ViT)  pre-trained with self-supervision (DINO). Compared to standard Slot Attention, the paper shows that this is sufficient to take a step up in terms of visual complexity. The method is thus evaluated on real-world datasets such as PASCAL VOC and COCO, but also MOVi datasets that contain realistic objects. The tasks that are evaluated include object discovery and localization, but also unsupervised object segmentation and semantic segmentation, and this method performs better or on par with the state of the art. 

strength_and_weaknesses: ## Strengths
* This paper provides a significant finding: it is the first time that the line of work in object-centric learning (here, specifically Slot Attention) scales to real-world images.

* The “trick” that makes it work is insightful and simple, which is to reconstruct the output of an image encoder. Instead of a randomly initialized encoder, the authors find it important to use a pre-trained and frozen feature extractor trained with self-supervision (here, DINO). These findings may be very useful for future work in unsupervised object-centric learning.  

## Weaknesses
### Semantic vs. instance grouping
It is clear from all figures that this approach tends to group semantically related objects together in the same slot, which is likely due to the semantic nature of DINO. This is also evident from the fact that its features have been used off-the-shelf for unsupervised semantic segmentation in prior work. However, this finding collides with the objective of object-centric learning which is to represent by slots individual scene elements rather than semantic categories. 

### Encoder pre-training 
Although the paper contains an analysis the role of the image encoder, only different architectures have been considered and supervised vs DINO pre-training (Table 6). It is worth asking if all self-supervised pre-training strategies are equal for the tasks considered in this paper, and in particular, for scaling object-centric methods to real-world data. Hence, I encourage the authors to look into how generalizable this approach is across a number of pre-training strategies. Different pre-training objectives (e.g. image reconstruction MAE, MSN) could have a different effect than contrastive learning or the DINO objective.

### Feature reconstruction objective
While the feature reconstruction objective offloads the complexity of real images to a pre-trained encoder, this can potentially limit the generalizability of the method to data that is similar to the encoder pretraining stage. It could thus be interesting to evaluate on ClevrTex (Karazija et al., NeurIPS D&B 2021) which is a textured version of the Clevr dataset and has been shown to be already too complex for the original slot attention mechanism to work. As the content of these images differs considerably from ImageNet pre-training the domain-dependency of the proposed method can be evaluated.

clarity,_quality,_novelty_and_reproducibility: The paper is well written, clear and of high quality. The appendix contains extensive additional results and implementation details improving reproducibility. Source code is promised to be released upon acceptance.


summary_of_the_review: The paper can be seen in two ways. As an approach to bring object centric learning to real world data and as a task paper for unsupervised semantic segmentation. While the paper excels in the first aspect, it falls behind specialised existing approaches for the second. Overall, as the field of object-centric learning is struggling to make the transition to real data, the paper contains valuable insights that are interesting for the community. It is thus not necessary for this paper to improve over current unsupervised, task-specific approaches and I recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors follow in the line of recent improvements on the slot-attention architecture for unsupervised object discovery that reconstruct a frame not in the raw pixel space, but rather in a more structured space (e.g. flow for SaVi, depth for SaVi++ or feature space of a network which is (pre-)trained in a self-supervised way in STEVE, which is the closest work to this one). In particular, instead of reconstructing the image in the feature space of a VQ-VAE, as done in STEVE, they propose to reconstruct in the feature space of a ViT pre-trained with DINO. The resulting model seems to perform on-par with prior work on the task on object discovery (no direct comparisons are reported). In addition, they report evaluation on related tasks of unsupervised object localization and instance/semantic segmentation where the proposed methods also performs similarly to existing approaches. 
strength_and_weaknesses: Strengths:

The paper is well written ad is easy to follow. 

The propose approach is sound.

Reported results seem o be on par with prior work (though the most important comparisons are missing).

A minimal ablation study of the proposed architecture is provided.


Weaknesses:

The novelty is minimal as the proposed method is extremely close to STEVE, with the only significant difference being the source of self-supervised pre-training for the feature space in which the frames are reconstructed.

Experimental evaluation on the main task of object discovery is incomplete. In particular, the authors do not compare to the most relevant approaches (SaVi, SaVi++, STEVE), and instead only compare to much weaker versions of these methods. Moreover, even the existing comparisons are not fair, as the prospered method uses a stronger ViT backbone. Results with a ResNet backbone have to be reported instead.

Positioning is not valid, as the authors claim that unlike prior works that utilize motion cues, their approach does not require additional supervision. In particular, the authors claim that motion constitutes "additional information" and use the term "motion annotations" which is meaningless. Motion is a bottom-up signal which comes for free with videos, thus any method that only relies on motion cues is in fact fully unsupervised. Abstract, introduction, and related work section require a major update to correctly position the proposed approach with respect to prior work.

On a similar note, related work overview is incomplete. The authors do not cite or discuss Bao et al., CVPR'22 who proposed a fully unsupervised object discovery approach which capitalizes on motion cues and achieves strong results in the real world. The authors need to discuss with work and compare to it on the KITTI dataset.

Finally, the ablation analysis is not sufficient. In particular, it's not clear if the self-supervised approach used for pre-training is important (only DINO is reported). The role of the pre-training backbone architecture is unclear as well. The authors claim that "ResNet50 performs clearly worse than the ViT", but the results in Table 6 do not support such a strong claim (ResNet does much better on ARI, but worse on mBO, which is a non-standard metric). Moreover, unlike ViT, ResNet does not require a strong initialization to converge. Given that the pre-training algorithm and the back bone architecture are the two main differences to STEVE, I did not learn much from this paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written and is easy to follow.

Originality: the novelty is minimal, as explained above.

Reproducibility: sufficient details are provided and the authors promise to release the code.
summary_of_the_review: The authors propose an approach which is very similar to prior work (especially to STEVE) and fail to provide a convincing evaluation which would demonstrate the benefits of their approach. In particular, no comparison to the closest prior methods is reported and the ablation study ignores or fails to justify the most important factors in the approach. Moreover, prior work overview is incomplete and positioning is invalid. I recommend a major revision and a resubmission to a different venue.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Four experts reviewed this paper with mixed scores (2 accept, 1 borderline reject). AC does feel that this work takes a meaningful step towards real-world object-centric learning by introducing a few new techniques. The reviewers did raise some valuable concerns. The authors are encouraged to make the necessary changes in the camera-ready version.


justification_for_why_not_higher_score: 1. Still many limitations when evaluating real-world scenes. 
2.  The clarity needs to be further improved.
justification_for_why_not_lower_score: Reasonable good results on real-world scenes.
Decision: Accept: poster