A Clustering Baseline for Object-Centric Representations

Authors: Federico Baldassarre, Josselin Somerville Roberts, Huy V. Vo, Maxime Oquab, Piotr Bojanowski
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: object-centric representations, self-supervised learning

Abstract:
Object-centric learning aims to discover and represent visual entities as a small set of object embeddings and masks, which can be later used for downstream tasks. Recent methods for object-centric learning build upon vision foundation models trained with self supervision because of the rich semantic features they produce. However, they often involve additional training to optimize for object mask accuracy for a specific granularity of objects on a test dataset, while overlooking the evaluation of the quality of the object embeddings which is arguably more important.

In this work, we demonstrate how to discover objects and parts with a simple multi-scale application of k-means to the features of an off-the-shelf backbone. Our method is fast and flexible, produces interpretable masks, preserves the quality of the backbone embeddings, does not require additional training, and can capture different part/whole structures.

We evaluate the quality of the obtained representation on a variety of downstream tasks including scene classification and action recognition in videos, showing that it surpasses the performance of fine-tuned object-centric learning methods. Object masks produced by our method also effectively capture real-world objects and parts at various granularity, with comparable quality to specialized methods when evaluated on unsupervised segmentation benchmarks. These results suggest rethinking the current approach to object-centric learning, with a greater focus on the quality of the representation.

Decision: reject
Reviewer scores: 3, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729716227643

thread_type:
official_review

<thread_object>
creator: reviewer_srbe

<content>
summary:
This paper introduces an object-centric framework for extracting object representations without the need for training. The proposed approach involves applying simple k-means clustering to features from off-the-shelf backbones, specifically DINOv2 [1]. The centroids from clustering, computed with different numbers of clusters (k values), are stored to capture varying levels of granularity. The resulting set of centroids is then treated as the object-centric representation of the image. These representations are evaluated on downstream tasks such as classification and video action recognition to assess their quality, and on image segmentation to determine how well they visually align with objects.

---
[1] Maxime Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, arXiv

strengths:
* Learning object-centric representations in an unsupervised manner is both crucial and challenging, making the task addressed in this paper highly significant.
* Evaluating object-centric representations beyond object segmentation, on downstream tasks like video action recognition, is an important demonstration of their usability. The evaluation uses diverse and comprehensive datasets.
* The paper is well-written and easy to follow.

weaknesses:
* Applying clustering to pretrained encoder features is a well-known concept. Previous work in object-centric learning, such as DINOSAUR [1] and SOLV [2], has reported k-means clustering results in their ablations. Furthermore, clustering DINO [3] features has been used for segmenting objects in videos [4] and images [5, 6, 7]. This paper currently provides an evaluation of this well-established clustering approach on different benchmarks, but it is unclear how the framework represents a significant improvement or introduces novelty.

* In section 3.2, assuming overlapping masks across different granularities is an advantage over current object-centric methods, it’s worth noting that SAM [8] can extract overlapping masks for any object at any scale with pixel-level precision. This approach offers higher precision than patch clustering and can be used to extract object-centric representations in a zero-shot manner. What is the advantage of the proposed model compared to SAM in this regard?

* Since the CLS token is included in the $S_N$ representation set, it is expected to perform similarly in single-object classification tasks (e.g., ImageNet). For multi-object datasets like COCO, centroids should enhance performance by adding more granularity, whereas the CLS token summarizes the entire scene. Therefore, the comparison with the CLS token in Table 1 doesn’t provide much insight. Moreover, the approach uses multiple levels of granularity ($k = \\{1, 2, 4, 8\\}$), whereas SPOT [9] is limited to a single scale (7, 8, or 16), placing SPOT at a disadvantage. For a fair comparison, SPOT should include multi-scale representations, similar to the evaluations in Tables 2 and 3.

* For video action recognition (Figure 3), how does dropping some portion of tokens in the “all-patches” setting affect the tradeoff between token count and performance? Redundant tokens are common in video understanding tasks [10]. This comparison is important to highlight the efficiency of clustering centroids.

* The proposed approach leverages DINOv2 [11], which has been shown to significantly improve performance in object-centric tasks [2], while other models in Table 3 rely on the original DINO. Despite this advantage, both DINOSAUR and SPOT outperform the proposed method when using a single scale, i.e., a fixed number of slots/centroids, suggesting that the performance gain may primarily stem from the use of multi-scale representations. Given this, a multi-scale version of prior object-centric representations, such as a collection of DINOSAUR slots with $S = \\{1, 2, 4, 8\\}$, is expected to provide richer information than k-means clustering.

---
[1] Maximilian Seitzer et al., Bridging the Gap to Real-World Object-Centric Learning, ICLR 2023

[2] Görkay Aydemir et al., Self-supervised Object-centric Learning for Videos, NeurIPS 2023

[3] Mathilde Caron et al., Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021

[4] Georgy Ponimatkin et al., A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation, WACV 2023

[5] Paul Engstler et al., Understanding Self-Supervised Features for Learning Unsupervised Instance Segmentation, arXiv

[6] Gyungin Shin et al., Unsupervised Salient Object Detection with Spectral Cluster Voting, CVPRW 2022

[7] Xudong Wang et al., Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023

[8] Alexander Kirillov et al., Segment Anything, ICCV 2023

[9] Ioannis Kakogeorgiou et al., SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive, CVPR 2024

[10] Tengda Han et al., Turbo Training with Token Dropout, BMVC2022

[11] Maxime Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, arXiv

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730417326835

thread_type:
official_review

<thread_object>
creator: reviewer_pf52

<content>
summary:
This paper presents a simple and effective clustering-based method for obtaining object-centric feature representations from state-of-the-art ViT models trained with self-supervision. While existing object-centric learning methods are designed with specialized architectures or training techniques, they are limited to specific granularity and small flexibility. The proposed method takes patch features from state-of-the-art ViT models and applies multiple steps of the K-means algorithm to obtain a small set of cluster centroids that represent objects at different levels of granularity. The proposed method shows efficacy at downstream tasks such as image and video classification, with much lower computational cost compared with the baselines.

strengths:
In comparison with learning-based methods, the proposed method offers a set of advantages. 1) It doesn't require any training or fine-tuning to extract object features from the off-the-shelf models. 2) It also renders object features with different levels of granularity on multiple objects/parts with multiple running of the k-means algorithm. 3) The centroids of the extracted features hold semantics that can be applied to downstream tasks.

weaknesses:
1) There is a lack of interpretability and expandability of the "object-centric" features extracted from ViT models. Current ViT models DINO and DINOv2 take image patches as input and are optimized with self-supervised losses. The self-attention layers extract the features at the image patch level, not the object level.  Simply applying K-means clustering on patch features does not guarantee that these features stand for the objects in the images. Wondering about the rationale behind why these features are defined as "object-centric features"?

2) The whole framework of the proposed method relies on the "well-trained features" from ViT models. With some epochs of clustering, these features are used for downstream tasks. Despite its simplicity, I wonder if further steps involving fine-tuning could be taken to update the ViT backbone. This is meaningful as it can help refine the self-supervised ViT features to bring out more semantic information.

3) What is the motivation for choosing video action recognition as one of the experiment setups? To evaluate the semantics quality of object-centric features, it may be a good idea to evaluate on more segmentation tasks on video or image datasets.

questions:
1) Is it possible to evaluate more video/image-based segmentation tasks to show the quality of the semantics of the learned features?

2) What layer of self-attention to extract the patch tokens? Using keys, queries, or values?

3) Time and efficacy analysis: comparison with the baselines?

4) Ablation to the number of running clustering K?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709866362

thread_type:
official_review

<thread_object>
creator: reviewer_ziug

<content>
summary:
This paper reveals that directly applying the classic K-Means clustering algorithm to DINOv2 features can yield meaningful object-centric representations. To enhance flexibility, the authors propose a multi-round K-Means clustering pipeline that utilizes varying K values and hierarchical resolutions. Experiments demonstrate the characteristics of the object-centric embeddings captured through K-Means clustering and compare these results to those obtained from learning-based methods.

strengths:
- The motivation is clear, and the method is straightforward to understand.
- Extensive experiments and thorough analysis are conducted to explore the characteristics of embeddings obtained by applying K-Means on DINO V2 features, along with comparisons to those captured by learning-based methods.

weaknesses:
- The main weakness of this work is the lack of novelty. DINOv2 features are already well-established for their ability to be clustered to represent objects, as evidenced not only by the original figures in DINO but also by numerous studies (both learning-based and non-parametric) in the field of unsupervised segmentation that have explored this capability. Consequently, it is expected that a simple K-Means clustering algorithm can be applied directly to DINOv2 features to yield meaningful object-centric representations.
- K-Means clustering also offers limited flexibility, as the choice of the number of clusters K and the design of the hierarchical structure to achieve optimal performance are hyperparameters that vary per dataset. Furthermore, whether the origin information contained in the resulting object embeddings is closely tied to the specific downstream tasks being addressed, making it difficult to assert that maintaining full semantic information is a definitive advantage.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730731643409

thread_type:
official_review

<thread_object>
creator: reviewer_sd7c

<content>
summary:
This paper presents a simple method that clusters feature from the representation from existing backbones, i.e., K-means clustering based on the extracted feature. The K is a fixed value. Some experiments are conducted to claim that this simple clustering method performs better than many other methods requiring additional training.

strengths:
1. The presentation is good. The story is very clear to understand.
2. The method is simple, and simplicity means easy to be applied to downstream fields.

weaknesses:
1. Claiming a kind of generally learned representation is better needs numerous diverse experiments to fully evaluate the method from different perspectives.  The experiments, which include only single-label classification and unsupervised segmentation, are insufficient. I think this work just selects a few experimental settings supporting the desired conclusion.
2. Clustering based on the representation produced by existing backbones is a commonly used baseline in fields like few-shot and unspervised learning. There is no new information in this paper.

questions:
I recommend the authors delving deeper into the studied problem to reveal some insightful observations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732809426863

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response to the reviewers

comment:
We thank the reviewers for their constructive comments and actionable feedback. We address the common concern about novelty and technical contribution below, and provide individual responses to other reviewers' feedback in separate comments.

## Novelty and contributions
In this work, we do not aim to introduce a novel method for object-centric learning, e.g. a new architecture or loss formulation. Instead, our focus is to bring an important perspective to the object-centric learning literature and re-consider a simple approach based on clustering as a strong baseline.

Object-centric learning, at its core, seeks to learn a structured representation of visual scenes by decomposing images into segments (objects, parts, or groups of objects) and attaching to each segment an embedding that represents it. The goal is therefore two-fold: the decomposition into segments and their representation.

In recent years, the most prominent works in the object-centric learning literature [1-5] have been focusing on the unsupervised segmentation aspect, i.e. evaluating whether the predicted segments effectively match objects in annotated datasets. Maximising segmentation metrics such as FG-ARI and mIoU has led to the development of increasingly sophisticated techniques, involving multi-stage training and custom layers. On the other hand, the quality and usefulness of the learned representation has been greatly overlooked. Only a few works attempt to probe the learned representation with ad-hoc prediction tasks, but no established benchmark and common metric to track exist.

In our work, we re-establish this two-sided nature of object-centric learning, and treat the evaluation of learned object embeddings as a first-class citizen. We show that the embeddings learned by all recent object-centric methods produce underwhelming results on all tasks considered, often being surpassed by a global representation such as a single CLS token.
Then, we consider the embeddings produced by k-means clustering, an algorithm that is often presented as a weak baseline in object-centric learning papers, and demonstrate that if properly applied it can outperform all recent object-centric learning approaches, while being at the same time simpler, faster and more efficient.

Therefore, the main contribution of this paper is the novel perspective on what really matters in object-centric learning, and what is missing in contemporary works. Our results suggest rethinking the current approach to object-centric learning. On one hand, future research should pay more attention to the evaluation of the learned representation. On the other hand, our findings hint at a shift to first building object-aware general representation learning then extracting objects with simple methods like ours.

## Updated pdf revision following the rebuttal
We integrated the suggestions from the reviewers and updated the manuscript according to their feedback. We uploaded two revisions that are identical in content, but the first revision highlights all modifications in orange color for ease of review. We summarize the changes below:
- Added a sentence in the abstract and in the introduction to better clarify novelty and contributions (reviewers `sd7c` and `ziUG`)
- Added references mentioned in the discussion (reviewer `srbe`)
- Additional experiments on representation quality in table 1 to evaluate the multi-scale representation of an ensemble of SPOT models (reviewer `srbe`)
- Extended and clarified the comparison between DINOv1 and DINOv2 in the representation learning evaluation in table 1 and unsupervised segmentation evaluation in table 2 (reviewers `pf52` and `srbe`)
- Additional datasets for the unsupervised segmentation experiments in table 2: ClevrTex, ADE20K (reviewer `pf52`)
- Ablation study in the appendix: video action recognition using a random selection of patches as the video representation (reviewer `srbe`)
- Backbone comparison for SPOT in the appendix: training SPOT on top of DINOv1 vs DINOv2 (reviewer `srbe`)
- Ablation study in the appendix: use query/key/values from the last self-attention layer instead of output patch features (reviewer `pf52`)

## Individual comments and discussion
We address all the points raised by the reviewers in individual comments under each review, providing clarifications, context, and additional experiments as needed. We encourage the reviewers to follow-up on our answers and look forward to the ensuing discussion.

## References
- [1] Object-Centric Learning with Slot Attention, Locatello et al.
- [2] Bridging the Gap to Real-World Object-Centric Learning, Seitzer et al.
- [3] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers, Kakogeorgiou et al.
- [4] Object discovery and representation networks, Hénaff et al.
- [4] Object-Centric Slot Diffusion, Jiang et al.
- [5] Illiterate DALL-E Learns to Compose, Singh et al.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a simple method that clusters features from existing backbones for obtaining object-centric feature representations. The manuscript was reviewed by four experts in the field. The recommendations are (3 x "3: reject, not good enough", "5: marginally below the acceptance threshold"). The reviewers raised many concerns regarding the paper, e.g., limited technical novelty, unclear motivation, unconvincing experimental evaluations, inadequate literature reviews, etc. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.