Title: HASSOD: Hierarchical Adaptive Self-Supervised Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: self-supervised learning

object detection
Abstract: The human visual perception system demonstrates exceptional capabilities in learning without explicit supervision and understanding the part-to-whole composition of objects. Drawing inspiration from these two abilities, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), a novel approach that learns to detect objects and understand their compositions without human supervision. HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image. Furthermore, HASSOD identifies the hierarchical levels of objects in terms of composition, by analyzing coverage relations between masks and constructing tree structures. This additional self-supervised learning task leads to improved detection performance and enhanced interpretability. Lastly, we abandon the inefficient multi-round self-training process utilized in prior methods and instead adapt the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. Through extensive experiments on prevalent image datasets, we demonstrate the superiority of HASSOD over existing methods, thereby advancing the state of the art in self-supervised object detection. Notably, we improve Mask AR from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B. Project page: https://HASSOD-NeurIPS23.github.io.

HASSOD: Hierarchical Adaptive Self-Supervised Object Detection

Abstract

The human visual perception system demonstrates exceptional capabilities in learning without explicit supervision and understanding the part-to-whole composition of objects. Drawing inspiration from these two abilities, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), a novel approach that learns to detect objects and understand their compositions without human supervision. HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image. Furthermore, HASSOD identifies the hierarchical levels of objects in terms of composition, by analyzing coverage relations between masks and constructing tree structures. This additional selfsupervised learning task leads to improved detection performance and enhanced interpretability. Lastly, we abandon the inefficient multi-round self-training process utilized in prior methods and instead adapt the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. Through extensive experiments on prevalent image datasets, we demonstrate the superiority of HASSOD over existing methods, thereby advancing the state of the art in self-supervised object detection. Notably, we improve Mask AR from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B. Project page: https://HASSOD-NeurIPS23.github.io .

1 Introduction

The development of human visual perception is remarkable for two key abilities: 1) Humans begin learning to perceive objects in their environment through observation alone [25], without needing to learn the names of these objects from external supervision. 2) Moreover, human perception operates in a hierarchical manner, enabling individuals to recognize the part-to-whole composition of objects [2, 23]. These characteristic capabilities offer valuable insights into the learning processes of object detectors, which still heavily rely on the availability and quality of fine-grained training data. For example, the state-of-the-art detection/segmentation model, Segment Anything Model (SAM) [18], is developed on a dataset of 11 million images and 1 billion object masks. It remains an open question how to effectively learn to detect objects and recognize their compositions from even larger-scale datasets ( e.g ., LAION-5B [26]) without such object-level annotations.

In prior work on self-supervised object detection [37, 38], a two-stage discover-and-learn paradigm is adopted: 1) Self-supervised visual representations [5, 15] are obtained, and a saliency-based method is employed to extract the most prominent one or few objects. 2) Subsequently, an object detector is trained based on these pseudo-labels, sometimes involving multiple rounds of self-training for refinement. However, despite such attempts to eliminate the need for external supervision, several weaknesses persist in these approaches: 1) Narrow coverage of objects. The focus on only one or few objects per image in previous methods undermines their ability to fully exploit the learning signals in

Input

CutLER

HASSOD (Ours)

HASSOD (Ours)

LVIS

Objects365

SA-1B

SOLO

SOLO

SOLO

TELEPEAJE

TELEPEAJE

TELEPEAJE

TAL

TAL

natural scene images containing dozens of objects, such as those in the MS-COCO dataset [20]. This narrow focus also restricts the capability of these methods to accurately detect and segment multiple objects within an image. 2) Lack of composition. Prior work often overlooks the composition of objects, neglecting the identification of hierarchical levels for whole objects, part objects, and subpart objects ( e.g ., considering an image of a bicycle; the bicycle is a whole object, its wheels and handles are parts, and the spokes and tires are subparts). This oversight not only limits the interpretability of learned object detectors, but also hinders the model's ability to tackle the intrinsic ambiguity in the task of segmentation. 3) Inefficiency. The reliance on multi-round self-training in earlier methods can result in inefficient and non-smooth training processes, which further constrains the potential of self-supervised object detection and comprehension of object composition.

Inspired by the unsupervised, hierarchical human visual perception system, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), aiming to address these limitations mentioned above and better harness the potential of self-supervised object detection, as depicted in Figure 1. First, unlike previous methods that limit their focus to one or few prominent objects per image, HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks, based on self-supervised visual representations. By adjusting the threshold for terminating the clustering process, HASSOD is capable of effectively determining the appropriate number of objects per image, thus better leveraging the learning signals in images with multiple objects.

The second key component of HASSOD is its ability to identify hierarchical levels of objects in terms of composition. By analyzing the coverage relations between masks and building tree structures, our approach successfully classifies objects as whole objects, part objects, or subpart objects. This novel self-supervised learning task not only improves detection performance, but also enhances the interpretability and controllability of the learned object detector, a property that prior self-supervised detectors lack. Therefore, HASSOD users can comprehend how the detected whole objects are assembled from smaller constituent parts. Simultaneously, they can control HASSOD to perform detection at their preferred hierarchical level, thereby catering more effectively to their needs.

Finally, HASSOD abandons the multi-round self-training used in previous methods which lacks efficiency and smoothness. Instead, we take inspiration from the Mean Teacher [22, 31] framework in semi-supervised learning, employing a teacher model and a student model that mutually learn from each other. This innovative adaptation facilitates a smoother and more efficient training process, resulting in a more effective self-supervised object detection approach.

In summary, the key contributions of HASSOD include:

A hierarchical adaptive clustering strategy that groups regions into object masks based on selfsupervised visual representations, adaptively determining the number of objects per image and effectively discover more objects from natural scenes.

The ability to identify hierarchical levels of objects in terms of composition (whole/part/subpart) by analyzing coverage relations between masks and building tree structures, leading to improved detection performance and enhanced interpretability.

A novel adaptation of the Mean Teacher framework from semi-supervised learning, which replaces the multi-round self-training in prior methods, leading to smoother and more efficient training.

State-of-the-art performance in self-supervised object detection, enhancing Mask AR from 20.2 to 22.5 on LVIS [11], and from 17.0 to 26.0 on SA-1B [18]. Remarkably, these results are achieved through training with only 1 / 5 of the images and 1 / 12 of the iterations required by prior work.

2 Related Work

Unsupervised object detection/discovery. Identifying and locating objects in images without using any human annotations is a challenging task, as it requires learning the concept of objects from image data without any external supervision. OSD [33] formulates this task as an optimization problem on a graph, where the nodes are object proposals generated by selective search, and the edges are constructed based on visual similarities. rOSD [34] improves the scalability of OSD with a saliency-based region proposal algorithm and a two-stage strategy. LOD [35] formulates unsupervised object discovery as a ranking optimization problem for improved computation efficiency. Following the observation that DINO [5], a self-supervised pre-training method, can segment the most prominent object in each image, LOST [29], FOUND [30], and FreeSOLO [37] train object detectors using saliency-based pseudo-labels. TokenCut [39] and CutLER [38] also use self-supervised representations, but generate pseudo-labels by extending Normalized Cuts [28]. Saliency-based region proposal and Normalized Cuts are both focused on the prominent objects in each image, and usually only propose one or few objects per image. Different from these approaches, HASSOD produces initial pseudo-labels using a hierarchical adaptive clustering strategy, which can adaptively determine the number of objects depending on the image contents.

Object detection by parts. Detecting objects by identifying their composing parts has been widely studied in computer vision. Deformable Parts Model (DPM) [9] is a seminal approach that utilizes discriminatively trained part-based models for object detection, which effectively models complex object structures and improves over monolithic detectors. A following method [6] not only detects the objects, but also simultaneously represents them using body parts, highlighting the importance of both holistic models and part-based representations. This idea is extended by leveraging both whole object and part detections to infer human actions and attributes [10], suggesting the advantage of a combined approach. In this work, we revisit this classic idea of representing and detecting whole objects as well as their parts in the context of self-supervised learning.

3 Approach

In this section, we introduce the learning process in our proposed approach, Hierarchical Adaptive Self-Supervised Object Detection (HASSOD). Following prior work on unsupervised object detection [29, 30, 37-39], HASSOD adopts a two-stage discover-and-learn process to learn a selfsupervised object detector, as illustrated in Figure 2. In the first stage, we discover objects from unlabeled images using self-supervised representations, and generate a set of initial pseudo-labels. Then in the second stage, we learn an object detector based on the initial pseudo-labels, and smoothly refine the model by self-training. The first stage is based on pre-trained, fixed visual features, and the second stage learns an object detector to improve over the fixed visual features and pseudo-labels. In the following subsections, we describe the three core components of HASSOD in detail.

3.1 Hierarchical Adaptive Clustering

In the first stage, HASSOD creates a set of pseudo-labels as the initial self-supervision source. We propose a hierarchical adaptive clustering strategy to discover object masks as pseudo-labels, using only unlabeled images and a frozen self-supervised visual backbone. Figure 3 provides an overview of this procedure. Our hierarchical adaptive clustering algorithm extends agglomerative

Stage 2: Object Detector Learning

(Sections 3.2 & 3.3)

Stage 1: Initial Pseudo-Label Discovery

(Section 3.1)

Unlabeled

Raw Images

Frozen

DINO ViT

Initial Pseudo-

Labeled Masks

Object Detector

Mean Teacher

Self-Training

DINO

ViT

Patch-Level

Features

Merge

𝜃

!

"#$%#

= 0.2

𝜃

&

"#$%#

= 0.4

𝜃

'

"#$%#

= 0.1

Merge

Merge

Post-

Process

Post-

Process

Post-

Process

All Initial

Pseudo-Labels

Ensem-

ble

Whole

Part

Subpart

Split

Hierarchical

Levels of Objects

clustering [12], grouping adjacent image patches into semantically coherent masks based on the similarity of self-supervised visual representations. More specifically, we use a frozen ViT-B/8 model [8] pre-trained on unlabeled ImageNet [7] by DINO [5], a self-supervised representation learning method, to extract visual features. For each image, we take the feature map generated by this model at its final Transformer layer [32]. Each spatial element in the feature map corresponds to a 8 × 8 patch in the original image.

To initiate the hierarchical adaptive clustering process, we treat each patch as an individual region. We then compute the pairwise cosine similarity between the features of adjacent regions to measure their closeness in the semantic feature space. The regions are gradually merged into masks that represent objects by iteratively performing the following steps: 1) Identify the pair of adjacent regions with the highest feature similarity. 2) If the similarity is smaller than the pre-set threshold θ merge , stop the merging process. 3) Merge the two regions, and compute the feature of the merged region by averaging all the patch-level features within it. 4) Update the pairwise similarity between this newly merged region and its neighbors. This merging process is visualized in Figure 3, columns 1-3.

Once the merging process is complete, we perform a series of automated post-processing steps to refine and select the masks, including Conditional Random Field (CRF) [19] and filtering out masks that are smaller than 100 pixels or contain more than two corners of the image. These steps are based on standard practices in previous work [38] and require no manual intervention. Our hierarchical adaptive clustering strategy effectively groups regions into object masks based on self-supervised visual representations, adaptively determining the appropriate number of objects per image. In

images containing multiple objects with heterogeneous semantic features, the merging process stops earlier, resulting in a larger number of regions corresponding to different objects. Conversely, in highly homogeneous images, more regions are merged, leading to fewer object masks. This adaptive approach enables HASSOD to cover more objects for self-supervised learning, rather than being limited by one or a few prominent objects in each image in prior work [38, 39].

In practice, we are not restricted to one single fixed threshold θ merge to determine the stopping criterion for the clustering process. Instead, we find it beneficial to ensemble results from multiple ( e.g ., 3) pre-set thresholds { θ merge i } 3 i =1 . When the currently highest feature similarity reaches one of these thresholds, we record the derived object masks from the merged regions at that step. Utilizing multiple thresholds allows us to capture objects of various sizes and at different hierarchical levels of composition, enabling a more comprehensive coverage of objects in scene images. The postprocessing and ensemble are visualized in Figure 3, columns 4-5.

3.2 Hierarchical Level Prediction

In the following second stage, HASSOD learns an object detection and instance segmentation model, e.g ., Cascade Mask R-CNN [4], using the initial pseudo-labels generated in the first stage. By training on such pseudo-labels, the model learns to recognize common objects across different training images, and thus achieves enhanced generalization to images which the model has not seen during training.

In addition to the standard object detection objective, we aim to equip our detector with the ability to understand the hierarchical structure among objects and their constituent parts. In HASSOD, we incorporate the concept of hierarchical levels into object masks by leveraging the coverage relations between them. Formally, we say mask A is covered by mask B when three conditions are satisfied (with respect to a pre-set coverage threshold θ cover % ): 1) More than θ cover % of pixels in mask A are also in mask B. 2) Less than θ cover % of pixels in mask B are in mask A. 3) Mask B is the smallest among all masks satisfying the previous two conditions. Intuitively, if mask B covers mask A, it suggests that A is a part of B and B is at a higher level than A. If we consider A and B as tree nodes, A should be a child of B. Using all such coverage relations, we can construct a forest of trees that contain all masks in an image. Ultimately, the roots of all trees in this image are considered as 'whole' objects, their direct children are 'part' objects, and all the remaining descendants are 'subpart' objects. An example is shown on the right side of Figure 3.

After identifying the hierarchical levels of object masks in the pseudo-labels, we attach a new classification head to the object detector for level prediction, which classifies each predicted object as a whole object, a part object, or a subpart object. This new component enables HASSOD to model object composition effectively, resulting in improved object detection performance and enhanced interpretability compared with previous self-supervised object detection methods. The hierarchical level prediction head is added alongside the existing foreground/background classification head, box regression head, and mask prediction head. Subsequently, we train the object detector using the initial set of object mask pseudo-labels obtained from the hierarchical adaptive clustering process, as well as the additional level prediction task.

3.3 Mean Teacher Training with Adaptive Targets

Notably, the initial pseudo-labels derived in the first stage contain noise and are not perfectly aligned with real objects. To improve over such noisy pseudo-labels, prior work [37, 38] usually employs multi-round self-training to refine the model, i.e ., using a well-trained detector to re-generate pseudolabels and re-train a new detector. For the first time , HASSOD refines the object detector efficiently and smoothly by adapting the Mean Teacher learning paradigm [22, 31] from semi-supervised learning to the fully self-supervised setting.

Before introducing our innovative adaptation of Mean Teacher in the self-supervised setting, we first briefly summarize the mutual-learning process in Mean Teacher (see Figure 4). Mean Teacher employs two models, a teacher and a student, which learn from each other. The teacher takes weaklyaugmented, unlabeled images as input and provides detection outputs as training targets for the student. The student's weights are updated to minimize the discrepancy between its predictions and the targets given by the teacher on the same unlabeled images but with strong augmentation. In the semisupervised setting, the student receives supervision from two sources simultaneously. One source is the 'teacher-to-student' branch mentioned above, and the other is the 'label-to-student' branch

Label-to-Student Branch

Teacher-to-Student Branch

Unlabeled

Image Batch A

Unlabeled

Image Batch B

Teacher

Detector

Student

Detector

Teacher's

Predictions A

Initial

Pseudo-Labels B

Student's

Predictions A

Student's

Predictions B

𝐿

!"#$%"&

𝐿

'#("'

𝐿

!)!#'

Strong

Aug.

Weak

Aug.

Discover

Predict

Predict

EMA

!"#$%"&

↗

'#("'

↘

Predict

Supervise

where the student learns from images with ground-truth labels. Both branches compute standard detection losses ( e.g ., bounding box classification and regression), and the student is optimized to minimize the total loss. The teacher's weights are an exponential moving average of the student's weights, ensuring smooth and stable training targets.

In HASSOD, we do not have any labeled images from human supervision but instead utilize two sources of self-supervision. One source is the initial pseudo-labels obtained from our hierarchical adaptive clustering, which functions similarly to the labels-to-student branch in the semi-supervised setting. The other source is the detection predictions made by the teacher model, which corresponds to the teacher-to-student branch in Mean Teacher. Different from standard Mean Teacher, our method employs adaptive training targets, as we gradually adjust the loss weights for the two branches. This is because the initial pseudo-labels may not effectively cover all possible objects, while the teacher model will progressively improve as a better source of supervision. Consequently, during Mean Teacher self-training, we continuously decrease the loss weight α label for the branch that uses the initial pseudo-labels and increase the loss weight α teacher for the branch based on the teacher's predictions, following a cosine schedule.

4 Experiments

In this section, we conduct extensive experiments to evaluate HASSOD in comparison with previous methods. We first describe the training details and efficiency in Section 4.1. Section 4.2 introduces the datasets and metrics used for evaluation. Section 4.3 presents our main results of self-supervised object detection and instance segmentation. Section 4.4 provides some qualitative results and analysis. Section 4.5 conducts further experiments to verify the effects of each component in HASSOD. Additional quantitative and qualitative results are included in the appendix.

4.1 Data-Efficient and Computation-Efficient Training

We train a Cascade Mask R-CNN [4] with a ResNet-50 [13] backbone on MS-COCO [20] images. The backbone is initialized from DINO [5] self-supervised pre-training. We use both the train and unlabeled splits of MS-COCO, totaling to about 0.24 million images. Notably, this amount of images is only 1 / 5 of ImageNet used by prior work CutLER [38]. Compared with ImageNet-like [7] iconic images, images in MS-COCO are mostly captured in complex scenes containing multiple objects with diverse layouts and compositions. Therefore, each image offers richer learning resources for object detectors, enabling effective detector training with significantly fewer images . The whole

training process spans 40,000 iterations, taking about 20 hours on 4 NVIDIA A100 GPUs. The efficiency and smoothness introduced by the Mean Teacher self-training approach reduces the training iterations to 1 / 12 of that required by CutLER [38], highlighting the computation efficiency of our training strategy. Implementation details are included in Appendix J.

4.2 Evaluation Datasets and Metrics

We mainly conduct our experiments in a zero-shot manner on the validation sets of three benchmark datasets, namely Objects365 [27], LVIS [11], and SA-1B [18]. Given that self-supervised object detection methods, including HASSOD, do not utilize class labels as a form of supervision, we follow prior work [37-39] and evaluate these models as class-agnostic detectors, comparing them only against the bounding boxes and masks provided in the dataset annotations.

Objects365 [27] is a large-scale object detection dataset containing 365 object categories. The combined validation sets of Objects365 v1 and v2 include 80,000 images in total.

LVIS [11] is a dataset that features a wide variety of over 1,200 object classes, using the same images as MS-COCO [20]. LVIS v1.0 validation set has 19,809 images, each annotated with object masks for instance segmentation.

SA-1B [18] is a recent dataset that includes 11 million images and 1 billion fine-grained, modelgenerated object masks. SA-1B provides a more comprehensive coverage of all potential objects, facilitating a more robust evaluation of self-supervised object detectors. As SA-1B does not provide a validation split, we utilize a random subset of 50,000 images for our assessment.

In terms of evaluation metrics, we focus primarily on average recall (AR) rather than average precision (AP). The choice of AR over AP is motivated by the nature of the self-supervised task. In a dataset with a fixed number of classes, objects not labeled by humans - simply because they do not fall under the designated classes - may still be detected by a self-supervised detection model. Standard AP calculation would penalize such predictions as false positives, despite them being valid detections. In contrast, AR does not suffer from this issue, making it a more appropriate metric for our context. By prioritizing recall, we can more accurately assess the ability of our model to identify all relevant objects in an image, which aligns with the goal of the self-supervised object detection task. We evaluate AR based on both bounding boxes for object detection ('Box AR') and masks for instance segmentation ('Mask AR'). Appendix A discusses the evaluation metrics in detail.

4.3 Self-Supervised Detection and Segmentation

After we use HASSOD to train the object detection and instance segmentation model, Cascade Mask R-CNN, on MS-COCO images, we evaluate our model on Objects365, LVIS, and SA-1B datasets in a zero-shot manner, i.e ., no further training on these three datasets. The whole training and evaluation process is repeated for three times, and we report the mean performance for conciseness. The standard deviation of AR is less than 0 . 6 on all three datasets. Complete evaluation results are included in Appendix H.

We compare HASSOD with prior state-of-the-art self-supervised object detection methods, including FreeSOLO [37] and CutLER [38]. We also include results from SAM [18], the latest supervised classagnostic detection/segmentation model, to gain understanding of the gap between self-supervised and supervised models, and how HASSOD is effectively closing this gap. To be consistent with other models, we provide SAM with only the raw images but no bounding boxes or points as prompts. For prior methods FreeSOLO, CutLER, and SAM, we directly evaluate the publicly available model checkpoints on the given datasets. Considering that the number of ground-truth labels per image may be greater than 100, we allow all models to output up to 1,000 predictions per image.

As shown in the main results summarized in Table 1, HASSOD significantly improves the detection and segmentation performance over previous self-supervised models FreeSOLO and CutLER. On Objects365, we improve the Box AR by 3.2; on LVIS, we improve Box AR by 3.3, and Mask AR by 2.3. The most remarkable performance gain is observed on SA-1B . We improve Box AR from 18.8 to 29.0 (relatively +54%) and improve Mask AR from 17.0 to 26.0 (relatively +53%).

We gain improved recalls for objects of all scales, but small and medium-sized objects relatively benefit more than large objects . For instance, our AR S is 2.6 × as CutLER's AR S on SA-1B. It is worth noting that detecting small objects is intrinsically harder than large objects - even though the

labels in SA-1B are produced by the same SAM model, when the bounding box prompts are no longer available, SAM can only reach a 20.0 Mask AR for small objects. Meanwhile, we halve the performance gap between self-supervised CutLER and supervised SAM from 15.1 Mask AR S to 7.1 Mask AR S . By learning from hierarchical levels of object compositions, HASSOD more effectively captures small objects which are part of whole objects.

4.4 Qualitative Results

In this section, we analyze some qualitative results on images from LVIS. The visualization is shown in Figure 5. Qualitative results on other datasets are included in Appendix K.

As shown in the examples, our proposed HASSOD exhibits a more comprehensive coverage of all objects in complex scenes, compared with the previous state-of-the-art self-supervised object detection method CutLER [38]. This advantage originates from the pseudo-labels generated by our hierarchical adaptive clustering, which includes a proper number of candidate objects per image according to the image contents, rather than only focusing on a fixed number of objects. Furthermore, HASSOD can predict the hierarchical level of each detected object. Being a fully self-supervised model, HASSOD has surprisingly gained the human-like ability to comprehend the composition of objects. This ability leads to better interpretability and controllability: Users of HASSOD can understand the composition of each object detected by the model. Meanwhile, users can also control the segmentation granularity by selecting the predictions at the desired hierarchical level.

The qualitative results also show some limitations of HASSOD. By comparing the last two columns, it can be observed that HASSOD produces relatively fewer predictions for 'subpart' objects. This is due to the distribution imbalance in the initial pseudo-labels, in which only about 10% objects are subparts. Also, the hierarchical levels learned by HASSOD are sometimes inconsistent with human perception. For example, instead of recognizing the person as a whole object in the last example image, HASSOD detects the upper and lower parts of the body as whole objects. Due to the lack of human supervision, the hierarchical prediction of HASSOD is not always aligned with humans. We further analyze this limitation in Appendix I.

Input

CutLER

HASSOD (Ours)

Whole Objects

HASSOD (Ours)

HASSOD (Ours)

fg 99%

fg 97%

g100%

94%

anse

anse

6

anse

6

fg1

4.5 Ablation Study

In this section, we conduct an ablation study to understand the effects of each component in HASSOD. To evaluate the performance more robustly against a larger set of human-annotated object-level labels, we combine the annotations of MS-COCO [20] and LVIS [11] on the val2017 split, because they are complementary to each other: LVIS uses the same images as MS-COCO, but labels more object classes. However, LVIS annotations are not as exhaustive as MS-COCO, meaning that objects within LVIS categories may not be labeled on all images. After combining the two sets of annotations and removing duplicates, there are around 20 object-level labels per image. We use this combined dataset for evaluation in all the ablation study experiments, unless otherwise specified.

Quality of initial pseudo-labels. We first examine how the design choices in our hierarchical adaptive clustering influence the quality of initial pseudo-labels. The results are summarized in Table 2. Each threshold θ merge i ∈ { 0 . 1 , 0 . 2 , 0 . 4 } leads to a different trade-off between the number of labels per image and the recall. A higher threshold stops the merging process earlier, and thus results in more pseudo-labels. With post-processing, we can improve pseudo-label quality by removing about half of the labels, and increase AP significantly without losing much AR. Finally, the ensemble of multiple merging thresholds θ merge i ∈ { 0 . 1 , 0 . 2 , 0 . 4 } brings the best overall pseudo-label quality. Appendices D, E, and F present more details regarding the choice of θ merge , computation costs, and ViT backbones in this stage.

Effects of hierarchical level prediction and Mean Teacher. After generating the initial pseudolabels with hierarchical adaptive clustering, we train the object detector with several techniques, including hierarchical level prediction, Mean Teacher self-training, and adaptively adjusting learning targets. The contribution of each technique is summarized in Table 3. Each component brings an additional 0.3 - 0.5 Mask AR improvement, and when they function together, the best overall performance can be achieved.

Improvement over initial pseudo-labels. Although the initial pseudo-labels are produced by a frozen DINO backbone and they tend to be noisy and coarse, HASSOD is not upper-bounded by the quality of the fixed initial pseudo-labels or the pre-trained backbone. This is because of the following reasons: 1) While the initially discovered pseudo-labels are noisy, in the learning stage we train a detection model to learn common objects and their hierarchical relations for enhanced generalization to unseen images. By learning this detector, we boost the detection AR and AP from 8.9 and 1.7 (the last row in Table 2) to 20.6 and 6.1 (the second row in Table 3), respectively. Meanwhile, the pre-trained backbone features are adapted for the detection task in an end-to-end manner. 2) We further leverage Mean Teacher for continual self-enhancement, and gradually minimize the negative impact of noisy initial pseudo-labels . The evolving teacher detector and its features provide improved pseudo-labels to the student. Notably, we can directly read out the predictions from the teacher as the refined hierarchical pseudo-labels, instead of inefficiently running the clustering algorithm using the enhanced backbone. Consequently, we further improve the detection AR and AP to 22.4 and 6.3 (the last row in Table 3), respectively.

5 Conclusion

We present Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), an approach inspired by human visual perception that learns to detect objects and understand object composition in a selfsupervised manner. HASSOD uses a hierarchical adaptive clustering strategy to propose a varying number of objects per image, and learns hierarchical levels of objects by analyzing geometric relations between objects. Mean Teacher self-training with adaptive targets facilitates the detector training process with smooth learning objectives and improved training efficiency. Empirical evaluation on recent large-scale image datasets Objects365, LVIS, and SA-1B demonstrates our significant improvement over prior self-supervised detectors. We detail the limitations and broader impacts of HASSOD in Appendix I.

Acknowledgments

This work was supported in part by the IBM-Illinois Discovery Accelerator Institute, NSF Grant #2106825, NIFA Award #2020-67021-32799, the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign through the NCSA Fellows program, the IllinoisInsper Partnership, and the Amazon Research Award. This work used NVIDIA GPUs at NCSA Delta through allocations CIS220014, CIS230012, and CIS230013 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program [3], which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296.

References

Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV , 2018. 15

Irving Biederman. Recognition-by-components: A theory of human image understanding. Psychological review , 94(2):115, 1987. 1

Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. ACCESS: Advancing innovation: NSF's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing , 2023. 11

Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In CVPR , 2018. 5, 6, 15, 16, 21

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021. 1, 3, 4, 6, 15, 18, 20, 21

Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR , 2014. 3

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR , 2009. 4, 6, 15, 20

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021. 4, 20

Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. TPAMI , 32(9):1627-1645, 2009. 3

Georgia Gkioxari, Ross Girshick, and Jitendra Malik. Actions and attributes from wholes and parts. In ICCV , 2015. 3

Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR , 2019. 3, 7, 8, 9, 14, 19

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction . Springer, 2009. 4

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. 6, 21

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV , 2017. 16

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022. 1

Tarun Kalluri, Weiyao Wang, Heng Wang, Manmohan Chandraker, Lorenzo Torresani, and Du Tran. Open-world instance segmentation: Top-down learning with bottom-up supervision. arXiv preprint arXiv:2303.05503 , 2023. 15

Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open-world object proposals without learning to classify. IEEE Robotics and Automation Letters , 7(2):5453-5460, 2022. 15

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV , 2023. 1, 3, 7, 8, 14, 17, 18, 19, 21, 23

Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected CRFs with Gaussian edge potentials. In NeurIPS , 2011. 4, 20

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014. 2, 6, 7, 9, 14, 15, 17, 20, 21

Yang Liu, Idil Esen Zulfikar, Jonathon Luiten, Achal Dave, Deva Ramanan, Bastian Leibe, Aljoša Ošep, and Laura Leal-Taixé. Opening up open world tracking. In CVPR , 2022. 15

Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In ICLR , 2021. 2, 5, 21

Stephen E Palmer. Hierarchical structure in perceptual representation. Cognitive psychology , 9(4):441-474, 1977. 1

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019. 21

Jean Piaget. The construction of reality in the child. Journal of Consulting Psychology , 19(1):77, 1955. 1

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In NeurIPS Datasets and Benchmarks Track , 2022. 1

Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV , 2019. 7, 8, 17, 19, 21, 22

Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. TPAMI , 22(8):888-905, 2000. 3

Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC , 2021. 3

Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín Vobeck` y, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Observing the background to discover objects. In CVPR , 2023. 3

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS , 2017. 2, 5

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. 4

Huy V Vo, Francis Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce. Unsupervised image matching and object discovery as optimization. In CVPR , 2019. 3

Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In ECCV , 2020. 3

Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsupervised object discovery. In NeurIPS , 2021. 3

Weiyao Wang, Matt Feiszli, Heng Wang, Jitendra Malik, and Du Tran. Open-world instance segmentation: Exploiting pseudo ground truth from learned pairwise affinity. In CVPR , 2022. 15

Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. FreeSOLO: Learning to segment objects without annotations. In CVPR , 2022. 1, 3, 5, 7, 8

Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In CVPR , 2023. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 20, 21

Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. TokenCut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. In CVPR , 2022. 3, 5, 7

Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https: //github.com/facebookresearch/detectron2 , 2019. 21

Appendix

In this appendix, Section A first explains the deficiency of traditional MS-COCO AP evaluation in the self-supervised setting and reasons for adopting AR on more class-extensively-annotated datasets like LVIS. Sections B and C address potential concerns about unfair comparison with CutLER [38] regarding the training data and the detector architecture. Sections D, E, and F present additional ablation study regarding the threshold choice, computation cost, and patch size of the ViT backbone in our hierarchical adaptive clustering algorithm. Section G studies different behaviors of SAM and our proposed HASSOD in detecting parts of objects. Sections H and K present more comprehensive quantitative and qualitative evaluation results for completeness. Section I provides the failure cases of HASSOD and analyzes its current limitations. Section J describes the detailed hyper-parameter and implementation setup.

A Deficiency of MS-COCO AP Evaluation in Self-Supervised Object Detection

Traditionally, the Average Precision (AP) metric on the MS-COCO dataset [20] has been a gold standard for assessing the performance of supervised object detection and instance segmentation models, which are trained and evaluated on a fixed set of object categories pre-defined by human annotators. However, we find this metric misleading in the context of self-supervised object detection , where class labels are not available to the model and class-agnostic predictions are necessary. In this work, we advocate for evaluating Average Recall (AR) on datasets with extensive class annotations ( e.g ., LVIS [11]) as a more reliable metric for comparing different methods. In this section, we discuss the inherent deficiencies of MS-COCO AP evaluation using an illustrative example.

To demonstrate this problem objectively, we compare two previous methods, CutLER [38] and SAM [18]. Both models function as class-agnostic object detectors, but SAM, trained with human supervision, evidently outperforms CutLER in terms of detecting and segmenting objects, as depicted in Figure 6. Surprisingly, when evaluated on MS-COCO annotations, SAM attains a mere 6.7 AP, which is approximately half of CutLER's 12.3 AP. In this case, the AP metric is contradicting with the observed performance of the two models in terms of accurately detecting and segmenting as many objects as possible.

Input

CutLER

12.3 AP on MS-COCO

SAM

6.7 AP on MS-COCO

The primary reason for this discrepancy lies in the annotations of MS-COCO. MS-COCO labels only 80 object categories, and model predictions are considered as true positives only if they fall within

these categories. Consequently, correct predictions for objects outside the 80 categories are unjustly deemed false positives and penalized by the AP metric, contradicting the objective of class-agnostic object detection and revealing the shortcomings of AP evaluation.

To correct these deficiencies in the evaluation metric, we need two changes: 1) Adopt a dataset with comprehensive annotations that include as many object categories as possible. If we substitute MS-COCO ground-truth annotations with LVIS, which labels over 1,200 object categories despite using the same images, the AP comparison is reversed: CutLER scores 4.5 AP on LVIS, while SAM attains a higher 6.1 AP. 2) Replace AP with the AR metric. Even with LVIS annotations, not all object categories are labeled in every image, resulting in certain valid object detection predictions still being penalized. Conversely, AR does not penalize such predictions and exhibits a more pronounced difference between CutLER and SAM (23.6 AR vs . 42.7 AR).

In summary, we advocate for AR on class-extensively-annotated datasets as the primary comparison metric, which can more accurately reflect the actual performance of class-agnostic object detectors with reduced bias. This choice of metric aligns with prior work on open-world detection [1, 17], segmentation [16, 36], and tracking [21] as well.

B Comparison of CutLER and HASSOD with Equal Training Data

As described in the main paper, HASSOD utilizes a ResNet-50 backbone initialized with DINO [5] weights pre-trained in a self-supervised manner on ImageNet [7], while subsequent training is conducted on MS-COCO [20] images. We choose MS-COCO for its compact size and richness in objects per image. Due to limited computation resources, we are unable to perform HASSOD training on ImageNet, and we leave large-scale training as one interesting future direction. This distinction in the training dataset may lead to concerns regarding the fairness of comparison with prior work, such as CutLER [38], which trains exclusively on ImageNet data. However, it is important to note that using both ImageNet and MS-COCO data in HASSOD does not grant HASSOD additional benefit compared with CutLER for two main reasons: 1) The two datasets are leveraged in separate stages and not in a blended manner. Specifically, ImageNet data are only used in the DINO pre-training stage, while MS-COCO is employed for detector training. 2) ImageNet contains approximately 5 × as many images as MS-COCO. Thus, if ImageNet is employed in the detector training stage, as is the case with CutLER, this would actually lead to a stronger detector.

In order to address these concerns more effectively and ensure an equal usage of training data , we conduct an additional experiment. Specifically, we train a Cascade Mask R-CNN [4] detector using the CutLER approach on MS-COCO images for one round (160,000 iterations), with the ResNet-50 backbone initialized with DINO pre-trained weights. We adopt CutLER's original implementation, with the sole modification being the use of MS-COCO images for training. We compare this model with a CutLER model trained on ImageNet for one round and our HASSOD model trained on MS-COCO. The evaluation is conducted against MS-COCO+LVIS annotations, as described in Section 4.5. The results are presented in Table 4.

By comparing the two CutLER models trained with MS-COCO and ImageNet, we observe that utilizing distinct datasets for backbone pre-training and detector training does not universally improve performance. The CutLER model, exclusively trained on ImageNet, surpasses its counterpart that uses

ImageNet for DINO pre-training and MS-COCO for detector training, exhibiting gains of 1.5 Mask AR and 0.4 Mask AP. Meanwhile, HASSOD, despite being trained on the smaller MS-COCO dataset and for a shorter duration, outperforms CutLER by 3.6 Mask AR . This remarkable performance is attributed to HASSOD's comprehensive object coverage in images and its efficient usage of training data.

C Additional Results on Mask R-CNN

In HASSOD, we train a Cascade Mask R-CNN [4] object detection and instance segmentation model. We choose this architecture following CutLER [38] and we ensure a fair comparison with this prior work. In fact, HASSOD can be applied on different detector architectures. As an example, we also train a Mask R-CNN [14] and compare its LVIS performance with CutLER in Table 5. With the same detector architecture, HASSOD produces a better detector than CutLER. More impressively, our Mask R-CNN (weaker architecture) outperforms CutLER's Cascade Mask R-CNN (stronger architecture) on LVIS , highlighting the advantage of our approach.

D Choosing Thresholds for Hierarchical Adaptive Clustering

When determining the merging thresholds { θ merge i } , we mainly consider our computational constraints and empirical observations. The decision of merging thresholds is not made based on validation performance (Table 2), ensuring that HASSOD is fully self-supervised.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Merging threshold

0

20

40

60

80

100

Number of pseudo-labels per image

Guidance by number of pseudo-masks. Our choice for { θ merge i } is primarily guided by the number of pseudo-label masks produced per image. Figure 7 shows the relationship between the number of masks per image and different thresholds. When θ merge ≥ 0 . 5 , the number of masks per image escalates rapidly. This steep increase incurs significant computational costs, both during the initial generation of pseudo-labels and the subsequent data loading and pre-processing procedures during model training. To strike a balance between computational efficiency and the desired mask granularity, thresholds of { θ merge i } = { 0 . 1 , 0 . 2 , 0 . 4 } are chosen.

Threshold generalization across datasets. Another noteworthy observation is the generalizability of these thresholds across various datasets. In Table 6, we present the number of generated pseudo-labels per image on three datasets. With the merging threshold θ merge fixed, the number of generated labels is relatively stable, regardless of the source image dataset. Therefore, our pre-set thresholds are generalizable and require no further tuning when transferred to other datasets. Meanwhile, our detection model was trained on MS-COCO images with pseudo-labels generated using our pre-set { θ merge i } , and could generalize well to other datasets in a zero-shot manner, as shown in Table 1. This fact shows that the thresholds { θ merge i } are effective regardless of evaluation datasets.

E Computational Costs in Hierarchical Adaptive Clustering

We adopt the hierarchical adaptive clustering strategy to generate our initial pseudo-labels. In this section, we provide additional details regarding computation costs in this procedure.

As shown in Figure 3, the hierarchical adaptive clustering contains four steps 'merge,' 'post-process,' 'ensemble,' and 'split.' Among them, the 'merge' step accounts for the major computation costs. We can analyze its time complexity: Suppose we have n patches in the beginning, then there are at most n merging steps before stopping. Each merging step requires retrieving the most similar pair of adjacent regions. The collection of adjacent pairs has size at most O ( n ) , and each operation requires time O (log n ) if this collection is organized as a balanced binary tree. Therefore, the merging process has time complexity O ( n log n ) . In our practice, the input image has resolution 480 × 480 , so n = 480 8 × 480 8 = 3 , 600 . This time complexity is affordable.

More concretely, we list the time costs in the hierarchical adaptive clustering on our computation platform in Table 7. Since the procedure is learning-free, we can parallelize the processing of images using more than one worker. On our computation nodes equipped with 4 NVIDIA A100 GPUs, we can reduce the processing time to 1.59 sec/image. With 4 such nodes, we can complete the pseudo-label generation for MS-COCO train and unlabeled splits (0.24 million images) in 1 . 59 × 0 . 24 × 10 6 4 × 86400 ≈ 1 day.

F Impact of Patch Size in Hierarchical Adaptive Clustering

We use the DINO [5] pre-trained ViT-B/8 backbone to generate the initial pseudo-labels through our hierarchical adaptive clustering. We observe that the small patch size 8 × 8 leads to better pseudo-label quality. In Table 8, we compare the mask quality of pseudo-labels generated by ViT-B/8 (patch size 8 × 8 ) vs . ViT-B/16 (patch size 16 × 16 ). For a fair comparison, we use 480 × 480 input resolution for ViT-B/8 and 960 × 960 for ViT-B/16, so that they have the same number of initial patches. With the same merging threshold θ merge , ViT-B/16 leads to slightly fewer labels per image, and the quality is significantly worse than ViT-B/8, especially for small and medium objects. Therefore, we apply ViT-B/8 in our experiments for its localized visual features and subsequent high-quality pseudo-labels.

G Different Behaviors of SAM and HASSOD in Object Part Detection

Segment Anything Model (SAM) [18], a supervised segmentation model, has demonstrated its ability in creating high-quality, fine-grained segmentation of images. Directly comparing the performance between SAM and HASSOD would be unbalanced, considering SAM requires extensive humanlabeled images for training, while HASSOD operates entirely under self-supervision. Despite this, it remains intriguing to understand their different behaviors. In this section, we delve into a qualitative comparison between SAM and our proposed HASSOD approach, paying particular attention to their respective abilities to detect constituent parts of whole objects. Figure 8 presents a visualization for this comparison.

Both SAM and HASSOD can perform fine-grained segmentation within complex scenes, successfully detecting individual object parts that constitute a whole entity. However, a clear distinction arises in their respective approaches towards the segmentation of these object parts. For scenes in which objects follow a grid pattern, where the whole entity is partitioned by regular boundary lines into semantically similar pieces, SAM tends to perceive each grid as a distinct object. Conversely, HASSOD adheres to a holistic perspective for such cases, grouping the grid parts together even at the subpart level, due to their semantically analogous features.

In particular, HASSOD excels at distinguishing object parts that contain different contents within whole objects. This skill of HASSOD is especially advantageous in certain real-world applications. For example, in medical imaging, HASSOD's fine-grained segmentation can potentially assist in identifying and separating different tissues or anatomical structures within a scan. Additionally, in the manufacturing industry, HASSOD could be used in quality control to check and compare individual components of an assembled product. While both segmentation strategies of SAM and HASSOD are valid, HASSOD's unique ability to distinguish semantically different parts within objects provides distinct advantages in a wide range of practical scenarios. Comprehensively quantifying and analyzing such an ability is crucial for advancement of self-supervised object detection and segmentation approaches, and we consider it as an important future direction.

Input

SAM

HASSOD (Ours)

Whole Objects

HASSOD (Ours)

g76%

H Additional Evaluation Results for Completeness

As an addition to Table 1 in the main paper, we present a comprehensive evaluation of HASSOD on the Objects365 [27], LVIS [11], and SA-1B [18] datasets in Table 9. For a robust measure of performance, we incorporate the standard deviation, estimated from three independently trained models.

I Limitations and Broader Impacts

Limitations. Due to the self-supervised nature of HASSOD, the learned object hierarchical levels may not be perfectly aligned with human perception. This mismatch may lead to over-segment or under-segment of objects in real-world applications.

Input

CutLER

HASSOD (Ours)

Severins-

Severins-

brucke

K

brucke

Koelnmesse

belnmesse

Im Sionstal→

P

Im Sionstal→

Im Sionstal

We observe that HASSOD performs unsatisfactorily in certain scenes due to this lack of human supervision. Figure 9 provides a visualization of the failure cases. In the first row, HASSOD mistakenly treats the lower parts of the two players as a single coherent object. Due to their similar color and texture, multiple overlapping instances of the same class can sometimes be perceived as one object. In the second row, HASSOD fails to predict a mask that encompasses the entire motorcycle. This object consists of highly heterogeneous parts, making it challenging to recognize them as components of a single entity. In the third row, HASSOD fails to detect the text 'Koelnmesse,' and the boundaries for other text are not clear. We also observe that such errors are not unique to HASSOD; they appear in prior self-supervised object detection methods like CutLER as well. We believe that further human supervision would be necessary for correcting these mistakes.

Broader impacts. Detecting object parts with self-supervision may be beneficial to real-world applications including robotic manipulation and inspection. As a general object detection method, we share risks associated with applying recognition models such as abuse of surveillance systems.

J Hyper-Parameters and Implementation Details

In the initial pseudo-label generation process, we use a frozen ViT-B/8 model [8] pre-trained on unlabeled ImageNet [7] by DINO [5], a self-supervised representation learning method, to extract visual features of train and unlabeled images in MS-COCO [20]. Following prior work CutLER [38], we resize the resolution of each image to 480 × 480 , leading to 60 × 60 patches as initial regions. The merging process stops at three thresholds θ merge 1 = 0 . 4 , θ merge 2 = 0 . 2 , θ merge 3 = 0 . 1 , and results from these three thresholds are ensembled after post-processing. The post-processing steps include Conditional Random Field (CRF) [19] and filling the holes in each mask. We also filter out low-quality masks that 1) have an Intersection-over-Union (IoU) smaller than 0.5 before and after CRF, 2) are smaller than 100 pixels, or 3) contain more than two corners of the image (which are likely

background). These post-processing steps are also used in prior work including CutLER [38]. After ensembling results from the three thresholds { θ merge i } 3 i =1 , we identify the hierarchical levels of each object mask based on the coverage relation analysis. The coverage threshold is set to θ cover % = 90% .

In the detector training stage, we train a Cascade Mask R-CNN [4] with a ResNet-50 [13] backbone on the same MS-COCO [20] images. The backbone is initialized from DINO [5] self-supervised pre-training. Our hyper-parameter setting for Mean Teacher mostly follows the practice of Unbiased Teacher [22]. The whole training process starts with a 'burn-in' stage, during which the student model is only trained on the initial pseudo-labels with a fixed learning rate 0.01 and fixed loss weights. After the burn-in stage, the teacher model is introduced, and we gradually adjust the learning rate from 0.01 to 0, the loss weight in the label-to-student branch from 1.0 to 0.0, and the loss weight in the teacher-to-student branch from 2.0 to 3.0, all following a cosine schedule. The whole training process spans 40,000 iterations with a batch size of 16 images. The training is performed on 4 × NVIDIA A100 GPUs. Our code is developed based on PyTorch [24] and Detectron2 [40].

K Additional Qualitative Results

In this section, we present visualization of detection results by CutLER [38] and HASSOD on Objects365 [27] in Figure 10 and SA-1B [18] in Figure 11 (see next pages). Qualitative results on LVIS has been included in the main paper, Figure 5.

Input

CutLER

HASSOD (Ours)

Whole Objects

HASSOD (Ours)

farm

farum

farm

SES

ML

ML

ML

Sunny Mart

Sunny Mart

Sunny Mart

Input

CutLER

HASSOD (Ours)

Whole Objects

HASSOD (Ours)

cecccccccco

PK-LGY

PK-LGY

PK-LGY

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a self-supervised object detection method (in fact, it can also do instance segmentation). This is based on an idea of bottom-up merge by strong feature representation like from DINO feature. The image patch grouping is hierarchical because of setting different stopping thresholds. This makes the model interpretable. Overall, the idea is simple and straightforward. The model can be trained by teacher-student framework. Experiments show better results than recent work MaskCut.\n\nSTRENGTHS:\n- the bottom-up merging method to solve the self-supervised object detection might be new.\n- Figure 1 and 2 are good to understand the method.\n- experiment results look good.\n\nWEAKNESSES:\n- The writing is bad. Many critical details are omitted which makes it super hard to under the method.\n- Some inconsistencies in the paper, needs to clarify.\n\nQUESTIONS:\nSome critical details are missing in the paper and they need to be clarified. 1) What is learned, the whole DINO backbone or just a head network? 2) For the Mean-Teacher, what are the losses used to supervise the student networks training? \n\nThe grouping process in section 3 is iterative and seems to be too slow. What is the time complexity?\n\nLine 138, how to ensemble?\n\nLine 195, in section 3, looks like only the feature representation is used, why train a Cascade mask rcnn here? Obviously, Cascade mask rcnn is not self-supervised.\n\nLine 233 and 234 are inconsistent with section 4.2. What exactly Objects365, Lvis, SAM are for evaluation or training?\n\nLvis should have a LVIS-rare benchmark.\n\nWhole, part, subpart should be given rigorous definitions.\n\nFor the limitations 2) and 3) in the second paragraph, I don't understand how they are solved after reading the whole paper.\n\nLIMITATIONS:\nThe discussion at the end of the paper is good.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a self-supervised object detection approach by employing a self-supervised pre-trained model (i.e., DINO) to hierarchically and adaptively group regions into object masks using multiple pre-defined thresholds based on cosine similarity in the feature space. The authors then adapt the Mean Teacher framework to train a student object detector with the initial pseudo-labels from the clustering process as well as the progressively refined pseudo-labels from the teacher model. Extensive experiments demonstrate the superiority of the proposed approach over existing self-supervised object detection and instance segmentation methods.\n\nSTRENGTHS:\n-\tThe experiments are extensive and the results are promising.\n-\tThe paper is generally well-written and easy to follow.\n\nWEAKNESSES:\n-\tThe authors use a set of pre-defined thresholds (i.e., {0.1, 0.2, 0.4}) to merge region features. However, considering different datasets may have different distributions, such COCO-tuned hyper-parameters may not be suitable when merging region features for other datasets. Thus, the authors have to select the optimal thresholds for each dataset when performing clustering, which limits the practical applications of the proposed method to some extent.\n-\tThe authors use some post-processing techniques like CRF to refine the masks. I am curious about the mask quality without such post-processing. Will the performance be dropped significantly?\n-\tAlthough the proposed method can segment an image with hierarchy, it seems that the mask quality is somewhat inferior to CutLER according to the qualitative results as shown in the paper.\n-\tThe authors choose DINO with ViT-B/8 model, i.e., each spatial element in the feature map corresponds to a 8x8 patch in the original image. What is the effect of the patch size? It seems that the patch size tends to affect the performance significantly and should also be ablated.\n-\tThe authors use Cascade Mask R-CNN to verify the effeteness of the proposed method. What about other kinds of detectors? It would be better to provide results based on more detectors to verify the generality of the proposed method.\n\nQUESTIONS:\nSee questions mentioned above. I am currently leaning towards borderline reject and hope the authors could address my concerns during the rebuttal.\n\nLIMITATIONS:\nThe authors have well addressed the limitations and the broader impacts in Sec. 5, which looks good to me.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis goal of this paper is to improve the performance of self-supervised object detection by enhancing the detection pseudolabels. In addition to semantic masks provided as detection labels to an object detector, the authors utilize a rule-based approach to automatically generate a hierarchical structure of objects and provide it as targets to the detector. By training on hierarchical pseudolabels, the detector learns about the compositional structure of objects and therefore performs better. Additionally, the authors use a teacher-student network to train the detector instead of multi-round self-training approach. The proposed approach outperforms two other self-supervised detection approaches.\n\nSTRENGTHS:\n- The authors tackle an important problem in perception; the hierarchical structure of objects. By providing object labels at multiple levels in a tree structure, the detector can learn patterns in compositional structures of objects (subparts - parts - whole), which helps improve the predictions. For example, a model can learn to condition a whole object prediction on the existence of its parts and subparts.\n- This hierarchical structure is not only found in vision, but also other perceptual modalities and NLP. Therefore, understanding how to generate and utilize this structure can be helpful in other tasks and across other modalities.\n- This method is completely self-supervised, based on features extracted from DINO trained on ImageNet. The improvement in detection performance comes at no supervision cost, only additional computational cost to generate the pseudolabels.\n- The approach shows promising quantitative results, surpassing other self-supervised methods.\n\nWEAKNESSES:\n- This approach, like many others, will always be upper bounded by the quality of the features and the robustness of the self-supervised representation learning approach used to generate them. The generation of labels depends on frozen features, so the detector will be sensitive to the quality of those features.\n- Approaches that use hierarchical clustering lose representation quality as they naively average high dimension feature representations to merge clusters. Therefore these methods cannot cluster the object representations to generate class labels for training. This is similar to approaches like TW-FINCH [1] that takes the same route for event segmentation. This being said, I'm interested in seeing whether high-level representations of objects generated by averaging low-level pixel/patch representation would provide good matching to true class labels. An alternative would be to add a (non-)linear on the average-based representations to classify the objects.  \n- While there is no supervision cost to generate the extra hierarchical labels, there is computational cost in clustering and structure discovery. It is important to share the computational cost in order to determine whether this approach is practical and can be scaled up.\n- I am not convinced that simple thresholding of the similarity metric provides accurate separation between the hierarchy levels. Using an arbitrary threshold will guarantee a tree structure but not necessarily a part-whole structure. The goal is not to find any tree structure whith parent masks covering children masks. These rules do not guarantee a part-whole structure. Each entity in the tree should be a coherent object (part-whole tree).\n- One suggestion is to plot $\\theta^{merge}$ against the number of objects in the image and detect regions with lowest slopes to be the thresholds. The intuition is that in these regions the model is less sensitive to changes in $\\theta$, therefore there could be good separation between hierarchical levels at these thresholds.\n- How do the authors decide on $\\theta^{merge}$? Is it using the ablations in Table 2? Table 2 is generated using groundtruth masks. So if the decision on $\\theta^{merge}$ is based on evaluating the masks with ground truth masks, this would be a significant flaw in the experimental protocol. It would violate the self-supervision claim. Usually researchers decide based on a small validation set, and even then it is still arguably supervised. This is an important point.\n\n[1] Sarfraz, Saquib, et al. \"Temporally-weighted hierarchical clustering for unsupervised action segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nQUESTIONS:\n- When using a single threshold to generate the masks, my understanding is that we should only get one level of objects. In other words, the output masks should not intersect with any other masks. How are the numbers in Table 2 generated when using a single $\\theta^{merge}$ threshold? This approach should only be possible when having intersecting masks (i.e., ensemble).\n\nLIMITATIONS:\n- Limitations about not aligning hierarchical levels with human perception are addressed. But it's missing limitations concerning the inability of this model to generate representations useful to classification of the detected boxes/masks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hierarchical Adaptive Self-Supervised Object Detection (HASSOD), an approach that  learns to detect objects and understand their compositions without human supervision. \nHASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image.\nHASSOD identifies the hierarchical levels of objects in terms of compositionality, by analyzing coverage\nrelations between masks and constructing tree structures.\nThe proposed method adapts the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process.\n\nThe Mask AR is improved from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B.\n\nSTRENGTHS:\nGood results of the process, exhibited  in Fig. 3.\n\nMany quantitative comparisons in Tables 1-3.\n\nNice to read the overall paper.\n\nGood to see the reference of the vital work of [7], even after more than a decade.\n\nAppendix Doc highlights the importance of a better metric than AR.\n\nWEAKNESSES:\nLot of experiments, results, but no analytics. Not even a single equation, cost function, hyper-parameter set mentioned. \n\nAdaptation of mean-teacher model, still makes it a self-supervised framework.,\n\nThe overall designed architecture is explained with examples of images, in a diagram in fig. 2.\nA flowchart would still be better.\nWhich part requires CNN-based learning, which part requires shallow learning (if any) or heuristics is\nnot so clear (except a bit in Sec. 4.1, a small 2nd para).\n\nOnly in the Appendix - \nThe platform/environment/resources used for implementation are specified.\nThe computational time required for each sub-stage of the process is also not mentioned,\nincluding the inference time. This is very important for practitioners.\n\nFew failure cases should have been highlighted with causes/reasons; Appendix is not thorough on this too.\n\nColor labels used for inference vs bounding box, often causes confusion - the color overlap for anyone to clearly understand.\n\nQUESTIONS:\nIf this proposed work is inline with self-supervision,\nwhat are the equivalent/compatible stages of  - \npretext learning using base data with no labels;\nfine-tuning with support set;\ntarget set for inference on downstream task?\n\nA table specifying the dataset distributions (although names given) and stages exploiting the same\nfor self-supervision should have been given.\n\nAny scope of few-shot learning or meta-learning framework in your proposed approach?\n\nLIMITATIONS:\nLack of analytics ;\nNO ablation studies with cost functions;\nLack of mapping to standard Self-supervision paradigm, is a concern.\n\nObject Detection in complex cases of overlap of objects, background clutter not provided.\n\nI think its high time that researchers also look at identification/detection in camouflage (a personal opinion).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}