HASSOD: Hierarchical Adaptive Self-Supervised Object Detection
Authors: Shengcao Cao
Dhiraj Joshi
Liangyan Gui
Yu-Xiong Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning
object detection
TL;DR: We present a fully self-supervised approach that learns to detect objects and understand their compositions.

Abstract: The human visual perception system demonstrates exceptional capabilities in learning without explicit supervision and understanding the part-to-whole composition of objects. Drawing inspiration from these two abilities, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), a novel approach that learns to detect objects and understand their compositions without human supervision. HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image. Furthermore, HASSOD identifies the hierarchical levels of objects in terms of composition, by analyzing coverage relations between masks and constructing tree structures. This additional self-supervised learning task leads to improved detection performance and enhanced interpretability. Lastly, we abandon the inefficient multi-round self-training process utilized in prior methods and instead adapt the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. Through extensive experiments on prevalent image datasets, we demonstrate the superiority of HASSOD over existing methods, thereby advancing the state of the art in self-supervised object detection. Notably, we improve Mask AR from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B. Project page: https://HASSOD-NeurIPS23.github.io.

HASSOD: Hierarchical Adaptive Self-Supervised Object Detection
Shengeao Cao'
Dhiraj Joshi Liang-Yas Yan Guil Yu-Xiong Wang'
University of Illinois at Urbana- Urbana-Champaign Champaign , (ouui,yw)@i1110 yww)@illinois.......m. edu edu ?djoshiuss ibm com
IBB Research
Abstract
The human visual perception system demonstrates exceptional capabilities in learn- ing without explicit supervision and understanding the part-to-whole composition of objects. Drawing inspiration from these two abilities, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), novel approach that learns to detect objects and understand their compositions without human supervi- sion. HASSOD employs hierarchical adaptive clustering strategy to group regions into object masks based sule-supe visual representations, adaptively de- termining the number objects per image. Furthermore, HASSOD identifies the hierarchical levels of objects in terms of composition, by analyzing coverage relations between masks and constructing tree structures. This additional self- supervised learning task leads improved detection performance and enhanced interpretability. Lastly, we abandon the inefficient multi-round self-training process utilized in prior methods and instead adapt the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. Through extensive experiments on prevalent image datasets, we demon- strate the superiority of HASSOD over existing methods, thereby advancing the state of the in supe-supe object detection. Notably, we improve Mask AR from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-IB. Project page: https://HAsSDD: -Ne thib.h..... io.
Introduction
The development of human visual perception is remarkable for two key abilities: 1) Humans begin learning to perceive objects their environment through observation alone [25], without needing to learn the names of these objects from external supervision. 2) Moreover, human perception operates in a hierarchical manner, enabling individuals to recognize the part-to-whole composition of objects [2, 23]. These characteristic capabilities offer valuable insights into the learning processes of object detectors, which still heavily rely on the availability and quality of fine-grained training data. For example, the state-of-the-an detecttionssegmenttiion model, Segment Anything Model (SAM) [18], is developed on a dataset of 11 million images and billion object masks. It remains an open question how effectively learn detect objects and recognize their compositions from even larger-scale datasets (e.g., LAION-5B [26]) without such object-level annotations.
In prior work self-supervised object detection [37,38], two-stage discover-and-leumm paradigm is adopted: 1) Self-supervised visual representations [5, 15] are obtained, and a saliency-basee method employed extract the most prominent one few objects. 2) Subsequently, is trained based on these pscudo-labels, sometimes involving multiple rounds of self self-training for refinement. However, despite such attempts eliminate the need for external supervision, several weaknesses persist in these approaches: 1) Narrow coverage of objects. The focus on only one or few objects per image in previous methods undermines their ability to fully exploit the learning signals in
object detector
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

LVIS
Objects365
SA-1B
TALL
Input
CutLER
HASSOD (Ours)
HASSOD (Ours) Subpart
Whole Objects
Whole
Part
Fully self-supervised object detection and instance segmentation on prevalent image datasets. Our approach, HASSOD, demonstrates significant improvement over the previous state-of-the-an method, CutLER [38], by discovering more comprehensive range of objects. Moreover, HASSOD understands the thole object composition like humans do, while previous methods cannot.
Figure 1:
natural scene images containing dozens of objects, such as those in the MS-COCO dataset [20]. This narrow focus also restricts the capability of these methods accurately detect and segment multiple objects within an image. 2) Lack of composition. Prior work often overlooks the composition of objects, neglecting the identification of hierarchical levels for whole objects, part objects, and subpart objects (e-g., considering an image of a bicycle; the bicycle is a whole object, its wheels and handles are parts, and the spokes and tires are subparts). This oversight not only limits the interpretability of learned object detectors, but also hinders the model's ability to tackle the intrinsic ambiguity in the task of segmentation. 3) Inefficiency. The reliance on multi-round self-training in earlier methods can result in inefficient and non-smooth training processes, which further constrains the potential of self-super object detection and comprehension of object composition.
Inspired by the unsupervised, hierarchical human visual perception system, we propose Hierarchi- cal Adaptive Self-Supervised Object Detection (HASSOD), aiming to address these limitations mentioned above and better harness the potential of self-supervised object detection, as depicted in Figure 1. First, unlike previous methods that limit their focus to or few prominent objects per image, HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks, based on self-supervised visual representations. By adjusting the threshold for terminating the clustering process, HASSOD is capable of effectively determining the appropriate number of objects per image, thus better leveraging the learning signals images with multiple objects.
The second key component of HASSOD is its ability to identify hierarchical levels objects in terms of composition. By analyzing the coverage relations between masks and building tree structures, our approach successfully classifies objects as whole objects, part objects, or subpart objects. This novel self-supervised learning task only improves detection performance, but also enhances the interpretability and controllability of the learned object detector, property that prior self-supervised detectors lack. Therefore, HASSOD users can comprehend how the detected whole objects are assembled from smaller constituent parts. Simultaneously. they can control HASSOD to perform detection at their preferred hierarchical level, thereby catering more effectively to their needs.
Finally, HASSOD abandons the multi-round self seff-training used in previous methods which lacks efficiency and smoothness. Instead, we take inspiration from the Mean Teacher [22, 31] framework in sumi-super learning. employing a teacher model and a student model that mutually learn from each other. This innovative adaptation facilitates smoother and more efficient training process, resulting in a more effective self-supervised object detection approach.
In summary, the key contributions of HASSOD include:

---

A hierarchical udaptive clustering strategy that groups regions into object masks based on self- supervised visual representations, adaptively determining the number of objects per image and
effectively discover more objects from natural scenes.
The ability to identify hierarchical levels of objects terms of composition (whole/part/subpart) by analyzing coverage relations between masks and building tree structures, leading to improved
detection performance and enhanced interpretability.
A novel adaptation of the Mean Teacher framework from semi-supervised learning, which replaces the multi-round self-training in prior methods, leading to smoother and more efficient training. State-of-the- art performance in self-supervised object detection, enhancing Mask AR from 20.2 to 22.5 on LVIS [i1], and from 17.0 to 26.0 on SA-1B [18]. Remarkably, these results are achieved through training with only 1/5 of the images and 1/12 of the iterations required by prior work.
2 Related Work
Unsupervised object detection/discoveryy Identifying and locating objects in images without using any human annotations is challenging task, as it requires learning the concept of objects from image data without any external supervision. OSD [33] formulates this task as an optimization problem on graph, where the nodes are object proposals generated by selective search, and the edges are constructed based on visual similarities. rOSD [34] improves the scalability of OSD with a saliency- -based region proposal algorithm and a two-stage strategy. LOD [35] formulates unsupervised object discovery as a ranking optimization problem for improved computation efficiency, Following the observation that DINO [5], a self- supervisdd pre-training method, can segment the most prominent object in each image, LOST [29]. FOUND [30], and FrecSOLO [37] train object detectors using saliency-based pseudo-labels. TokenCut [39] and CuiLER [38] also use self-supervised representations, but generate pseudo-labels by extending Normalized Cuts [28]. Saliency-basee region proposal and Normalized Cuts both focused on the prominent objects in each image, and usually only propose one or few objects per image. Different from these approaches, HASSOD produces initial pseudo-labels using hierarchical adaptive clustering strategy, which can adaptively determine the number of objects depending on the image contents.
Object detection by parts. Detecting objects by identifying their composing parts has been widely studied in computer vision. Deformable Parts Model (DPM) [9] a seminal approach that utilizes discriminatively trained part-based models for object detection, which effectively models complex object structures and improves over monolithic detectors. A following method [6] not only detects the objects, but also simultaneously represents them using body parts, highlighting the importance of both holistic models and part-based representations. This idea is extended by leveraging both whole object and part detections to infer human actions and attributes [10], suggesting the advantage of combined approach. In this work, we revisit this classic idea of representing and detecting whole objects well their parts in the context of supervud learning.
Approach
In this section, we introduce the learning process in our proposed approach, Hierarchical Adap- tive Self-Supervised Object Detection (HASSOD). Following prior work unsupervised object detection [29, 30, 37-39], HASSOD adopts a two-stage discover-and-lead supervised object detector, as illustrated in Figure 2. In the first stage, we discover objects from unlabeled images using self-supervised representations, and generate a set of initial pseudo-labels. Then in the second stage, we learn an object detector based on the initial pseudo-labels, and smoothly refine the model by self-training. The first stage is based on pre-trained, fixed visual features, and the second stage learns object detector to improve the following subsections, we describe the three core components of HASSOD in detail.
process learn self-
the fixed visual features and pseudo-labels. In
3.1 Hierarchical Adaptive Clustering
In the first stage, HASSOD creates a set of pseudo-labels as the initial self-supervision source. We propose hierarchical adaptive clustering strategy to discover object masks as pseudo-labels, using only unlabeled images and frozen self-supervised visual backbone. Figure 3 provides an overview of this procedure. Our hierarchical adaptive clustering algorithm extends agglomerative
3

---

Stage Initial Lscudo-Label Discovery (Section 3.1)
Stage 2: Object Delector Learning (Sections 3.2 & 3.3)
Unisbeled Raw Images
Frowen DINO VIT
Inttil Peeexl-
Object Detector Coocnde Misk R-ONN
Labrled Masks
Mean Teather Seif-Training
Figure 2: Two-stage liscover-and-less process in HASSOD. Stage I uses frozen, self-supervised DINO [5] ViT backbone to discover initial pseudo-labels from unlabeled images. Stage 2 learns an object detector improve over the pre-trained features and initial pseudo-labels.
Hierarchical Levels of Objects
PatcheLovd
Mergo
Frataro
Parsiodabec
Mergo
Sclport
Figure 3: Hierarchical adaptive clustering and hierarchical levels objects. The procedure of creating initial pseudo-labels for training the object detector without any human annotations includes the following steps: (Initialize) Visual features are extracted from the given image by a ViT pre-trained with DINO [5], and each 8 x patch is initialized as one individual region. (Merge) Adjacent regions with the highest feature similarities are progressively merged into object masks, until the pre-set thresholds 0m are reached. (Post-Process) Object masks selected and refined using simple post-processing techniques. (Ensemble) Results from multiple thresholds [ " Ji=1 are combined to ensure better coverage of potential objects. (Split) Analysis of coverage relations divides objects into three hierarchical levels: whole, part, and subpart. The example the right illustrates the tree structure of object composition: The whole aircraft is composed of an upper and a lower part. The upper part further consists of left wing, right wing, and a person standing it.
clustering [12], grouping adjacent image patches into semantically coherent masks based on the similarity of sulf-supervised visual representations. More specifically, use frozen VIT-B/8 model [8] pre-trained on unlabeled ImageNet [7] by DINO [5], a self-supervised representation learning method, to extract visual features. For each image, take the feature map generated by this model at its final Transformer layer [32]. Each spatial element in the feature map corresponds to 8 x 8 patch in the original image.
To initiate the hierarchical adaptive clustering process, we treat each patch as an individual region. We then compute the pairwise cosine similarity between the features of adjacent regions to measure their closeness in the semantic feature space. The regions are gradually merged into masks that represent objects by iteratively performing the following steps: 1) Identify the pair of adjacent regions with the highest feature similarity. 2) If the similarity is smaller than the pre-set threshold (merge stop the merging process. 3) Merge the two regions, and compute the feature of the merged region by averaging all the patch-level features within 4) Update the pairwise similarity between this newly merged region and its neighbors. This merging process is visualized in Figure 3, columns 1-3.
Once the merging process complete, we perform a series of automated post-processing steps to refine and select the masks, including Conditional Random Field (CRF) [19] and filtering out masks that are smaller than 100 pixels or contain more than two corners of the image. These steps are based on standard practices in previous work [38] and require no manual intervention. Our hierarchical adaptive clustering strategy effectively groups regions into object masks based on self-supervised visual representations, adaptively determining the appropriate number of objects per image. In

---

images containing multiple objects with heterogeneous semantic features, the merging process stops earlier, resulting in a larger number of regions corresponding different objects. Conversely, in highly homogeneous images, more regions are merged, leading to fewer object masks. This adaptive approach enables HASSOD to cover more objects for self-supervised learning, rather than being limited by one or a few prominent objects in each image in prior work [38, 39].
In practice, we are not restricted to one single fixed threshold (merge to determine the stopping criterion for the clustering process. Instead, we find it beneficial to ensemble results from multiple (e.g., 3) pre-set thresholds (0_ar))) Ri-1. When the currently highest feature similarity reaches one of these thresholds, we record the derived object masks from the merged regions at that step. Utilizing multiple thresholds allows us capture objects of various sizes and at different hierarchical levels of composition, enabling more comprehensive coverage of objects in scene images. The post- processing and ensemble are visualized in Figure 3, columns 4-5.
3.2 Hierarchical Level Prediction
In the following second stage, HASSOD learns an object detection and instance segmentation model, e.g., Cascade Mask R-CNN [4], using the initial pseudo-labels generated in the first stage. By training on such peeudo-labess, the model learns to recognize common objects across different training images, and thus achieves enhanced generalization to images which the model has not seen during training.
In addition to the standard object detection objective, we aim to equip our detector with the ability to understand the hierarchical structure among objects and their constituent parts. In HASSOD, we incorporate the concept of hierarchical levels into object masks by leveraging the coverage relations between them. Formally, we say mask A is covered by mask B when three conditions are satisfied (with respect to a pre-set coverage threshold gewer))): 1) More than gowerr% of pixels in mask are also in mask B. 2) Less than genver% of pixels in mask B are in mask A. 3) Mask B is the smallest among all masks satisfying the previous two conditions. Intuitively, if mask B covers mask A, suggests that A is a part of B and B is at higher level than A. If we consider A and B as tree nodes, A should that contain all masks in an image. Ultimately, the roots of all trees in this image are considered as "whole" objects, their direct children are "part" objects, and all the remaining descendants are "subpart" objects. An example is shown on the right side of Figure 3.
child of B. Using all such coverage relations, we can construct a forest of trees
After identifying the hierarchical levels of object masks in the pseudo-labels, attach a new classification head to the object detector for level prediction, which classifies each predicted object as a whole object, a part object, or a subpart object. This new component enables HASSOD to model object composition effectively, resulting in improved object detection performance and enhanced interpretability compared with previous self-supervised object detection methods. The hierarchical level prediction head is added alongside the existing foreground/backgrounn regression head, and mask prediction head. Subsequently, we train the object detector using the initial set of object mask pseudo-labels obtained from the hierarchical adaptive clustering process, as well as the additional level prediction task.
classification head, box
3.3 Mean Teacher Training with Adaptive Targets
Notably, the initial lseudo-labels derived in the first stage contain noise and are not perfectly aligned with real objects. To improve over such noisy pseudo-labels, prior work [37, 38] usually employs multi-round self-training to refine the model, ie., using a well-trained detector to re-generate pseudo- labels and re-train new detector. For the first time, HASSOD refines the object detector efficiently and smoothly by adapting the Mean Teacher learning paradigm [22, 31] from semi-supervised learning to the fully self-supervised setting.
Before introducing our innovative adaptation of Mean Teacher in the self-supervised setting, we first briefly summarize the mutual-learnigg process in Mean Teacher (see Figure 4). Mean Teacher employs two models, a teacher and student, which learn from each other. The teacher takes weakly- augmented, unlabeled images as input and provides detection outputs as training targets for the student. The student's weights are updated to minimize the discrepancy between its predictions and the targets given by the teacher on the same unlabeled images but with strong augmentation. In the semi- supervised setting, the student receives supervision from two sources simultaneously. One source is the mancher-t--stdentt branch mentioned above, and the other is the Sabel-to-sttdemm branch
5

---

Saccher-ooSStudent Branch
Teacher's Predictions A
Predict
Wenk
Ang.
Desector
Lipacher
Unlabeled
Student's Predictions
Image Batch
EMA
Predict
Xifreacher
Strong Ang.
Stadlntt Desector
Supervise
Leceal
Image Batch
Unlabeled
Predict
Student's Predictions
Xiljabul
Luael
Initial
Diseover
Pscutio-Lbbels B
Label-to-Studde Branch
Figure 4: Mean Teacher self-training with adaptive targets in HASSOD. Two detectors of the same architecture, the teacher and the student, learn from each other to improve over the initial pseudo- labels. The teacher is updated as the exponential moving average (EMA) of the student. The student receives supervision from two branches: The teacher-to-student branch (top) encourages the student to mimic the teacher's predictions; the label-to-student branch (bottom) minimizes the discrepancy between the student's predictions and the initial pseudo-labels, During training, our proposed adaptive target strategy increases the weight for the teacher-to-studee branch, Oreacher and decreases the weight for the tooou-te-ttudent branch, alabll since the teacher becomes a more and more reliable self- superviieni source compared with the initial pseudo-labels.
where the student learns from images with ground-truth labels. Both branches compute standard detection losses (e.g., bounding box classification and regression), and the student is optimized to minimize the total loss. The teacher's weights are an exponential moving average of the student's weights, ensuring smooth and stable training targets.
In HASSOD, we do not have any labeled images from human supervision but instead utilize two sources of self- self-supervision. One source is the initial pseudo-labels labels obtained from our hierarchical adaptive clustering, which functions similarly to the tibels-to-studers branch in the comi-supervised setting. The other source is the detection predictions made by the teacher model, which corresponds to the teacher-to-studen branch Mean Teacher. Different from standard Mean Teacher, our method employs adaptive training targets, as gradually adjust the loss weights for the two branches. This is because the initial pseudo-labels may not effectively cover all possible objects, while the teacher model will progressively improve better source of supervision. Consequently, during Mean Teacher self-training, we continuously decrease the loss weight datell for the branch that uses the initial pseudo-labels labels and increase the loss weight O'eacher for the branch based the teacher's predictions, following a cosine schedule.
Experiments
In this section, conduct extensive experiments to evaluate HASSOD in comparison with previous methods. first describe the training details and efficiency in Section 4.1. Section 4.2 introduces the datasets and metrics used for evaluation. Section 4.3 presents main results of self-supervised object detection and instance segmentation. Section 4.4 provides some qualitative results and analysis. Section 4.5 conducts further experiments to verify the effects of each component in HASSOD. Additional quantitative and qualitative results are included the appendix.
4.1
Data-Efficient and Computation-Effiiien Training
We train a Cascade Mask R-CNN [4] with a ResNet-50 [13] backbone on MS-COCO [20] images. The backbone is initialized from DINO [5] superviedd pre- tre-training. We use both the train and unlabeled splits of MS-COCO, totaling to about 0.24 million images. Notably, this amount of images is only 1/5 of ImageNet used by prior work CutLER [38]. Compared with ImageNet-like [7] iconic images, images in MS-COCO are mostly captured in complex scenes containing multiple objects with diverse layouts and compositions. Therefore, each image offers richer learning resources for object detectors, enabling effective detector training with significantly fewer images. The whole

---

training process spans 40,000 iterations, taking about 20 hours on 4 NVIDIA A100 GPUs. The efficiency and smoothness introduced by the Mean Teacher self- ceff-training approach reduces the training iterations to 1/12 of that required by CutL.ER [38], highlighting the computation efficiency of our training strategy. Implementation details are included in Appendix
4.2 Evaluation Datasets and Metrics
We mainly conduct our experiments zero-shot manner on the validation sets of three benchmark datasets, namely Objects365 [27], LVIS [11], and SA-IB [18]. Given that self-supervised object detection methods, including HASSOD, do not utilize class labels form of supervision, we follow prior work [37-39] and evaluate these models as class-agnostic detectors, comparing them only against the bounding boxes and masks provided in the dataset annotations.
Objects365 [27] is a large-scale object detection dataset containing 365 object categories. The combined validation sets of Objects365 v1 and v2 include 80,000 images in total.
LVIS [11] is a dataset that features a wide variety of over 1,200 object classes, using the same images as MS-COCO [20]. LVIS v1.0 validation object masks for instance segmentation.
has 19,809 images, each annotated with
SA-1B [18] 3 is a recent dataset that includes 11 million images and billion fine-grained, model- generated object masks. SA-IB IB provides more comprehensive coverage of all potential objects, facilitating more robust evaluation of self-supervised object detectors. As SA-IB does not provide a validation split, we utilize random subset of 50,000 images for our assessment.
In terms of evaluation metries, we focus primarily on average recall (AR) rather than average precision (AP). The choice of AR over AP is motivated by the nature of the self-supervised task. In a dataset with fixed number of classes, objects not labeled by humans- - simply because they do not fall under the designated classes may still be detected by a self-supervised detection model. Standard AP calculation would penalize such predictions as false positives, despite them being valid detections. In contrast, AR does not suffer from this issue, making it more appropriate metric for our context. By prioritizing recall, we can more accurately assess the ability of our model identify all relevant objects in an image, which aligns with the goal of the self-supervised object detection task. We evaluate AR based on both bounding boxes for object detection ("Box AR") and masks for instance segmentation ("Maak AR"). Appendix A discusses the evaluation metrics in detail.
4.3 Self-Supervised Detection and Segmentation
After we use HASSOD to train the object detection and instance segmentation model, Cascade Mask R-CNN, on MS-COCO images, we evaluate our model on Objects365, LVIS, and SA-1B datasets in a zero-shot manner, i.e., no further training on these three datasets. The whole training and evaluation process is repeated for three times, and report the mean performance for conciseness. The standard deviation of AR is less than 0.6 on all three datasets. Complete evaluation results are included in Appendix H.
We compare HASSOD with prior sfate-of-the self-supervised object detection methods, including FreeSOLO [37] and CutLER [38]. We also include results from SAM [18], the latest supervised class- agnostic detection/segmeenttion model, to gain understanding of the gap between self supervupe and supervised models, and how HASSOD is effectively closing this gap. To be consistent with other models, we provide SAM with only the raw images but no bounding boxes or points as prompts. For prior methods FreeSOLO. CutLER and SAM, we directly evaluate the publicly available model checkpoints on the given datasets. Considering that the number of ground-truth labels per image may be greater than 100, we allow all models output up to 1,000 predictions per image.
As shown in the main results summarized in Table 1, HASSOD significantly improves the detection and segmentation performance over previous self-supe models FreeSOLO and CutLER. On Objects365, we improve the Box AR by 3.2; on LVIS, we improve Box AR by 3.3, and Mask AR by 2.3. The most remarkable performance gain is observed on SA-1B. We improve Box AR from 18.8 to 29.0 (relatively +54%) and improve Mask AR from 17.0 to 26.0 (relatively +53%).
We gain improved recalls for objects of all scales, but small and medium-sized objects relatively benefit more than large objects. For instance, our AR is 2.6x as CutLER's ARg on SA-1B. is worth noting that detecting small objects is intrinsically harder than large objects even though the
7

---

Table I: Comparison of self-supervised object detection and instance segmentation methods on prevalent image datasets. We consider the average recall (AR) instead of average precision (AP) as the main metric, because valid detection of objects outside the categories defined by human annotations is penalized by AP. HASSOD significantly outperforms the previously best methods FreeSOLO [37] and CutLER [38] in terms of AR at all object scales (Small. Medium, and Large). To understand the extent of improvements, we also include results from state-of-the-arr supervised model SAM [18]. HASSOD leads to reduced gap between fully selff-uuer models and supervised SAM. Notably, HASSOD only uses 1/5 of training images and 1/12 of training iterations as CutLER.
Box
Mask
Method
AR
ARg
ARM
AR
AP
AR
AR3
ARM
ARt AP
Objects365 [27]
SAM [18]
54.9
32.1
60.5
67.6
11.9
FreeSOLO [37]
10.2
0.2
5.8
23.4
3.4
No ground-truth mask
CutLER [38]
35.8
17.6
36.1
50.5
11.5
annotations Objects365
HASSOD (Ours)
39.0
21.4
40.4
52.1
11.0
LVIS [11]
SAM [18]
42,7
27.7
66.3
75.5
6.1
46.1
31.1
71.3
74.6
6.7
FreeSOLO [37]
6.4
0.3
9.7
34.6
1.9
5.9
0.2
9.2
31.7
1.9
CutLER [38]
23.6
13.1
36.2
55.6
4.5
20.2
11.3
31.1
46.2
3.6
HASSOD (Ours)
26.9
15.6
42.2
56.9
4.9
22.5
12.7
36.1
47.8
4.2
SA-/B [18]
SAM [18]
60.5
19.8
59.8
81.5
38.2
60.8
20.0
59.9
82.2
38.9
FreeSOLO [37]
2.4
0.0
0.1
7.4
1.5
2.2
0.0
0.2
6.9
1.5
CutLER [38]
18.8
5.1
14.6
32.8
9.0
17.0
4.9
13.9
28.5
7.8
HASSOD (Ours)
29.0
13.3
25.1
43.8
15.5
26.0
12.9
22.8
38.3
13.8
labels in SA-1B are produced by the same SAM model, when the bounding box prompts are no longer available, SAM can only reach 20.0 Mask AR for small objects. Meanwhile, we halve the performance gap between self-supervised CutLER and supervised SAM from 15,1 Mask ARs 7.1 Mask ARg. By learning from hierarchical levels of object compositions, HASSOD more effectively captures small objects which are part of whole objects.
4.4 Qualitative Results
In this section, we analyze some qualitative results on images from LVIS. The visualization shown in Figure 5. Qualitative results other datasets are included in Appendix K.
As shown in the examples, our proposed HASSOD exhibits more comprehensive coverage of all objects in complex scenes, compared with the previous state-of-the self-supervised object detection method CutLER [38]. This advantage originates from the pseudo-labels generated by our hierarchical adaptive clustering, which includes proper number of candidate objects per image according to the image contents, rather than only focusing on a fixed number objects. Furthermore, HASSOD can predict the hierarchical level of each detected object. Being fully self-supervised model, HASSOD has surprisingly gained the human-like ability to comprehend the composition of objects. This ability leads to better interpretability and controllability: Users of HASSOD can understand the composition of each object detected by the model. Mcanwhile, users can also control the segmentation granularity by selecting the predictions at the desired hierarchical level.
The qualitative results also show some limitations of HASSOD. By comparing the last two columns, it can be observed that HASSOD produces relatively fewer predictions for "subpart" objects. This is due the distribution imbalance the initial pseudo-labels, in which only about 10% objects are subparts. Also, the hierarchical levels learned by HASSOD are sometimes inconsistent with human perception. For example, instead recognizing the person as a whole object in the last example image, HASSOD detects the upper and lower parts of the body whole objects. Due to the lack of human supervision, hierarchical prediction of HASSOD not always aligned with humans. We further analyze this limitation in Appendix I.

---

LOTTO
Input
CutLER
HASSOD (Ours) Whole Objects
HASSOD (Ours)
HASSOD (Ours) Part Subpart
Whole Part
Figure 5: Qualitative results on LVIS images. Overall, our HASSOD successfully detects more objects compared with CutLER [38]. CutLER tends to detect only one few prominent objects in the image, while HASSOD captures other objects well (e.g., bread in row I, and traffic sign row 3). Moreover, HASSOD learns the composition of objects (e.g., cat-face-eye row 2, and vehicle-wheel-tire in row 4), which is similar to human perception.
4.5 Ablation Study
In this section, we conduct an ablation study to understand the effects of each component in HASSOD. To evaluate the performance more robustly against larger set human-annotated object-level labels, we combine the annotations of MS-COCO [20] and LVIS [11 I] the va12017 split, because they are complementary to each other: LVIS uses the same images as MS-COCO, but labels more object not exhaustive MS-COCO, meaning that objects within LVIS categories may not be labeled on all images. After combining the two sets of annotations and removing duplicates, there around 20 object-level labels per image. We use this combined dataset for evaluation in all the ablation study experiments, unless otherwise specified.
classes. However, LVIS annotations
Quality of initial pseudo-labels. We first examine how the design choices in our hierarchical adaptive clustering influence the quality of initial pseudo-labels. The results are summarized in Table 2. Each threshold 0. E 10.1, 0.2, 0.4) leads to different trade-off between the number of labels per image and the recall. A higher threshold stops the merging process earlier, and thus results in more pseudo-labels. With post-processing. we can improve pseudo-label quality by removing about half of the labels, and increase AP significantly without losing much AR. Finally, the ensemble of multiple merging thresholds 0.tergee E {0.1, 0.2, 0.4} brings the best overall pseudo-label quality. Appendices D, E, and present more details regarding the choice (merge, computation costs, and ViT backbones in this stage.
Effects hierarchical level prediction and Mean Teacher. After generating the initial pseudo- labels with hierarchical adaptive clustering. we train the object detector with several techniques, including hierarchical level prediction, Mean Teacher self-training, and adaptively adjusting learning targets. The contribution of each technique is summarized in Table 3. Each component brings an additional 0.3 0.5 Mask AR improvement, and when they function together, the best overall performance can be achieved.

---

Table 2: Ablation study on factors influencing the quality of initial pseudo-labels. The threshold (marge controls the stopping criterion of the merging process, and affects AR and the number of pseudo-labels. Applying post-processing can remove low-quality pseudo-labels without decreasing AR by large margin. Ensemble of multiple pseudo-label sources leads the best overall quality.
Post-
Labels per Img
Mask
Method
(merge
Process
AR
ARg
ARM
AR
AP
MaskCut [38]
1.85
3.5
0.0
2.0
20.0
1.5
0.1
5.33
4.4
0.8
5.1
16.6
1.2
0.1
2.58
4.1
0.6
5.1
15.6
1.8
Hicrarchical adaptive
0.2
8.36
5.5
1.2
7.0
18.9
1.3
0.2
4.20
5.3
0.9
7.0
18.4
1.8
clustering
0.4
23.33
7.9
2.1
12.0
21.7
0.7
(Ours)
0.4
11.61
7.8
1.7
12.1
22.1
1.3
ensemble
12.69
8.9
1.7
12.4
29.1
1.7
Table 3: Ablation study factors influencing the training of the object detector. Hierarchical level prediction is introduced as an auxiliary task self-supervision. Mean Teacher self self-training replaces the vanilla multi-round self self-training and brings smoother and more efficient training process. The weights of two learning targets, initial pscudo-labels and teacher predictions, are adaptively adjusted to build more effective curriculum. All the three key designs combined for the best overall
performance of HASSOD.
Level Prediction
Mean Teacher
Adaptive Targets
Mask
AR
ARs
ARM
ARt
AP
20.2
9.2
30.4
42.5
5.7
20.6
9.7
30.1
43.9
6.1
22.1
10.9
32.9
44.5
5.5
22.4
11.3
32.9
45.0
6.3
Improvement over initial pseudo-labels. Although the initial pseudo-labels are produced by frozen DINO backbone and they tend be noisy and coarse, HASSOD not upper-bounded by the quality of the fixed initial pseudo-labels or the pre-trained backbone. This is because of the following reasons: 1) While the initially discovered pseudo-labels are noisy, in the learning stage we train detection model to learn common objects and their hierarchical relations for enhanced generalization to unseen images. By learning this detector, we boost the detection AR and AP from 8.9 and 1.7 (the last row in Table 2) to 20.6 and 1 (the second row in Table 3), respectively. Meanwhile, the pre-trained backbone features are adapted for the detection task in an end-to-end manner. 2) We further leverage Mean Teacher for continual self-enhance self-enhancement, and gradually minimize the negative impact of noisy initial pseudo-labels. The evolving teacher detector and its features provide improved pscudo-labels to the student. Notably, can directly read out the predictions from the teacher the refined hierarchical pseudo-labels, instead inefficienlly running the clustering algorithm using enhanced backbone. Consequently, further improve the detection AR and AP to 22.4 and 6.3 (the last row in Table 3), respectively.
Conclusion
We present Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), an approach inspired by human visual perception that learns to detect objects and understand object composition in self- supervised manner. HASSOD uses hierarchical adaptive clustering strategy to propose a varying number of objects per image, and learns hierarchical levels of objects by analyzing geometric relations between objects. Mean Teacher self self-training with adaptive targets facilitates the detector training process with smooth learning objectives and improved training efficiency. Empirical evaluation on recent large-scale image datasets Objects365, LVIS, and SA-1B demonstrates our significant improvement over prior self-supervised detectors. We detail the limitations and broader impacts of HASSOD in Appendix I.
10

---

Acknowledgments
This work was supported in part by the IBM-IIlinois Discovery Accelerator Institute, NSF Grant #2106825, NIFA Award 17200-67021-27999 the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaar Insper Partnership, and the Amazon Research Award. This work used NVIDIA GPUs NCSA Delta through allocations CIS220014, CIS230012, and CIS230013 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services Support (ACCESS) program [3], which is supported by NSF Grants #2138259, #21382866 #21 1833077, #2137603, and #2138296.
through the NCSA Fellows program, the Illinois-
References
[1] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran, Zero-shot object detection. ECCV, 2018. 15
[2] Irving Biederman. Recogition-byy-comoonntt A thcory of human image understanding. Psychological review, 94(2):115, 1987.
[3] Timothy Boemner Stephen Deems, Thomas Furlani, Shelley L. Knuth, and John Towns. ACCESS: Advancing innovation: NSF's advanced cyherinfrrutruucuuee coordination ecosystem: Services support. In Practice and Experience Advanced Research Computing, 2023. 11
[4] Zhaowei Cai and Nuno Vasconeelos. Cascade R-CNN: Delving into high quality object detection. In CVPR, 2018. 5, 6, 15, 16, 21
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In /CCV, 2021. 1,3, 4, 6, 15, 18, 20, 21
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing ohjects using holistic models and body parts. In CVPR, 2014.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database. CVPR, 2009. 4, 6, 15, 20
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 4, 20
[9] Pedro F Felzenszwalh, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part- based models. TPAMI, 3229):1677 2009. 3
[10] Georgia Gkioxari, Ross Girshick, and Jitendra Malik. Actions and attributes from wholes and parts. In ICCV, 2015. 3
[11] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019. 3, 7, 8, 9, 14, 19
[12] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2009, 4
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016. 6, 21
[14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 16
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[16] Tarun Kalluri, Weiyao Wang. Heng Wang. Manmohan Chandraker, Lorenzo Torresani, and Du Tran. Open-world instance segmentation: Top-down learning with bottom-up supervision. arXiv preprint arXiv:2303.00 055003 2023. 15
[17] Dahun Kim, Tsung Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open-world object proposals without learning to classify. IEEE Roboties and Automation Letters, 7(2):5453-544 2022. 15
11

---

[18] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023. 1, 3, 8, 14, 17. 18, 19, 21,21.3
[19] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected CRFs with Gaussian edge potentials. In NeurIPS, 2011. 4, 20
[20] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects context. ECCV, 2014. 2, 6, 7, 9, 14, 15, 17, 20, 21
[21] Yang Liu, Idil Esen Zulfikar, Jonathon Luiten. Achal Dave, Deva Ramanan, Bastian Leibe, Aljoša OSep. and Laura Leal-Taixé. Opening up open world tracking. In CVPR, 2022. 15
[22] Yen-Cheng Liu, Caih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In ICLR, 2021. 2, 5, 21
[23] Stephen Palmer. Hierarchical structure in perceptual representation. Cognitive psychology, 9(4):441 1977. 1
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zach Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-peeformmncce deep learning library. In NeurlPS, 2019. 21
[25] Jean Piaget. The construction of reality in the child. Journal of Consulting Psychology. 19(1):77, 1955. 1
[26] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open acage-scale dataset for training next generation image-text text models. In NewrIPS Datasets and Benchmarks Track, 2022. 1
[27] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019. 7, 8, 17, 19, 21, 22
[28] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. TPAMI, 22(8):888 905, 2000. 3
[29] Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC, 2021. 3
[30] Oriane Siméoni, Chloé Sekkat, Gilles Puy. Antonía Vobecky, Éloi Zablocki, and Patrick Pérez. Unsuper- vised object localization: Observing the background to discover objects. In CVPR, 2023. 3
[31] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NewrlPS, 2017. 5,5
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017. 4
[33] Huy V Vo, Francis Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce. Unsupervised image matching and object discovery as optimization. In CVPR, 2019. 3
[34] Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In ECCV, 2020. 3
[35] Van Huy Vo, Flena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsupervised object discovery. In NeurlPS, 2021. 3
[36] Weiyao Wang, Matt Feiszli, Heng Wang, Jitendra Malik, and Du Tran. Open world instance segmentation: Exploiting pseudo ground truth from learned pairwise affinity. In CVPR, 2022. 15
[37] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. FreeSOLO: Learning to segment objects without annotations. In CVPR, 2022. 1, 3, 5, 7,
[38] Xudong Wang. Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In CVPR, 2023. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 20, 21
12

---

[39] Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley. and Dominique Vaufreydaz. TokenCut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. In CVPR, 2022. 3, 5.7
[40] Yuxin Wu, Alexander Kirillov,, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https: http:///www.eeeoeeeeeeeeeeetttto 2019, 21
13

---

Appendix
In this appendix, Section A first explains the deficiency of traditional MS-COCO AP evaluation in the self-supervised setting and reasons for adopting AR on more class-xtenssivlly-meeee
datasets like LVIS. Sections and C address potential concerns about unfair comparison with CutLER [38] regarding the training data and the detector architecture. Sections D, E, and F present additional ablation study regarding the threshold choice, computation cost, and patch size the ViT backbone in our hierarchical adaptive clustering algorithm. Section G studies different behaviors of SAM and our proposed HASSOD in detecting parts of objects. Sections and K present more comprehensive quantitative and qualitative evaluation results for completeness. Section provides the failure cases of HASSOD and analyzes its current limitations. Section describes the detailed pepee-pprameter and implementation setup.
A Deficiency of MS-COCO AP Evaluation in Self-Supervised Object Detection
Traditionally, the Average Precision (AP) metric on the MS-COCO dataset [20] has been a gold standard for assessing the performance of supervised object detection and instance segmentation models, which are trained and evaluated on fixed set of object categories pre-defined by human annotators. However, we find this metric misleading in the context of self sepeesupeee object detection, where class labels are not available to the model and class-agnostic predictions are necessary. In this work, we advocate for evaluating Average Recall (AR) on datasets with extensive class annotations (e.g., LVIS [11]) as a more reliable metric for comparing different methods. In this section, we discuss the inherent deficiencies of MS-COCO AP evaluation using an illustrative example.
To demonstrate this problem objectively, we compare two previous methods, CutLER [38] and SAM [18]. Both models function as class-agnostic object detectors, but SAM, trained with human supervision, evidently outperforms CutLER in terms of detecting and segmenting objects, as depicted in Figure Surprisingly, when evaluated on which is approximately half of CutLER's ' 12.3 AP. In this case, the AP metric is contradicting with the observed performance of the two models in terms of accurately detecting and segmenting as many objects as possible.
annotations, SAM attains mere 6.7 AP,
Input
CutLER 12.3 AP on MS-C0C0
SAM
6.7 AP on MS-COCO
Figure MS-COCO average precision (AP) evaluation may not accurately reflect the perfor- mance of class-agnostic object detectors. We compare two previous class-agnostic object detec- tion/segmentation models, CutLER [38] and SAM [18]. Despite SAM, supervised model, detecting more objects with superior localization than CutLER, it achieves only half the AP of CutLER.
The primary reason for this discrepancy lies in the annotations MS-COCO. MS-COCO labels only 80 object categories, and model predictions are considered as true positives only if they fall within
14

---

these categories. Consequently, correct predictions for objects outside the 80 categories are unjustly deemed false positives and penalized by the AP metrie, contradicting the objective of class- agnsssic object detection and revealing the shortcomings of AP evaluation.
To correct these deficiencies in the evaluation metric, need two changes: 1) Adopt a dataset with comprehensive annotations that include as many object categories as possible. If we substitute MS-COCO ground-truth annotations with LVIS, which labels over 1,200 object categories despite using the same images, the AP comparison is reversed: CutLER scores 4.5 AP on LVIS, while SAM attains a higher 6. 1. AP. 2) Replace AP with the AR metric. Even with LVIS annotations, not all object categories are labeled in every image, resulting in certain valid object detection predictions still being penalized. Conversely, AR does not penalize such predictions and exhibits more pronounced difference between CutLER and SAM (23.6 AR vs. 42.7 AR).
In summary, we advocate for AR on blass-eteessiell--nneen -nnootated datasets the primary comparison metric, which can more accurately reflect the actual performance of class-agnostic object detectors with reduced bias. This choice of metric aligns with prior work on open-world detection [1, 17], segmentation [16, 36], and tracking [21] as well.
B Comparison of CutLER and HASSOD with Equal Training Data
As described in the main paper, HASSOD utilizes ResNet-50 backbone initialized with DINO [5] weights pre-trained in a self-supervised manner on ImageNet [7], while subsequent training is conducted MS-COCO [20] images. We choose MS-COCO for its compact size and richness in objects per image. Due to limited computation resources, we are unable to perform HASSOD training on ImageNet, and we leave large-scale training as one interesting future direction. This distinction in the training dataset may lead to concerns regarding the fairness of comparison with prior work, such as CutLER [38], which trains exclusively on ImageNet data. However, it is important to note that using both ImageNet and MS-COCO data in HASSOD does not grant HASSOD additional benefit compared with CutLER for two main reasons: 1) The two datasets are leveraged in separate stages and not in a blended manner. Specifically, ImageNet data are only used in the DINO pre-training stage, while MS-COCO is employed for detector training. ImageNet contains approximately 5x as many images as MS-COCO. Thus, if ImageNet is employed in the detector training stage, as is the case with CuiLER, this would actually lead a stronger detector.
Table 4: Comparison between CutLER [38] and HASSOD concerning the training image dataset. Although both CutLER and HASSOD leverage DINO [5] weights pre-trained on ImageNet, employ- ing MS-COCO images in the detector training stage does not lead to superior performance. Cutl .RR trained on ImageNet surpasses CutLER trained on MS-COCO across all metrics. Simultancously, HASSOD outperforms both CutLER models, despite using MS-COCO training data and requiring fewer training iterations.
Training
Training Iterations
Mask
Method
Images
AR
ARs
ARM
ARt
AP
CutLER [38]
MS-COCO [20]
160,000
17.3
6.2
24.2
46.1
5,8
CutLER [38]
ImageNet [7]
160,000
18.8
7.2
27.6
46.6
6.2
HASSOD (Ours)
MS-COCO [20]
40,000
22.4
11.3
32.9
45.0
6.3
In order to address these concerns more effectively and ensure an equal usage of training data, we conduct an additional experiment. Specifically, we train a Cascade Mask R-CNN [4] detector using the CutLER approach MS-COCO images for one round (160,000 iterations). with the ResNet-50 backbone initialized with DINO pre-trained weights. We adopt CutLER's s original implementation, with the sole modification being the use of MS-COCO images for training. We compare this model with a CutLER model trained on ImageNet for one round and our HASSOD model trained on MS-COCO. The evaluation is conducted against MS-COCO+LVIS annotations, described in Section 4.5. The results are presented in Table 4.
By comparing the two CutLER models trained with MS-COCO and ImageNet, we observe that utilizing distinct datasets for backbone pre-training and detector training does not universally improve performance. The CutLER model, exclusively trained on ImageNet, surpasses its counterpart that uses
15

---

ImageNet for DINO pre- vre-traning and MS-COCO for detector training, exhibiting gains of 1.5 Mask AR and 0.4 Mask AP. Meanwhile, HASSOD, despite being trained the smaller MS- MS-COCO dataser and for shorter duration, oulperforms CuiLER by 3.6 Mask AR. This remarkable performance is attributed HASSOD's comprehensive object coverage images and efficient usage of training data.
C Additional Results on Mask R-CNN
In HASSOD, we train a Cascade Mask R-CNN [4] object detection and instance segmentation model. We choose this architecture following CutLER [38] and ensure fair comparison with this prior work. In fact, HASSOD can be applied on different detector architectures. As an example, we also train Mask R-CNN [14] and compare its LVIS performance with CutLER in Table 5. With the same detector architecture, HASSOD produces a better detector than CutLER. More impressively, Mask R-CNN (weaker architecture) outperforms CutLER's Cascade Mask R-CNN (stronger architecture) on LVIS, highlighting the advantage of our approach.
Table 5: Comparison between CutLER [38] and HASSOD with different detector architectures on the LVIS dataset. When training models of the same architecture, HASSOD outperforms CuiLER (row vs. 2, row 3 vs. 4). Notably, HASSOD has a stronger AR even with weaker architecture (row 2 3) compared with CutLER.
Rox
Mask
Architecture
Method
AR
ARg
AR
AR₁
AP
AR
ARg
AR
AR
AP
CutLER [38)
20.7
10.4
33.3
52.0
4.1
18.5
9.6
29.1
44,9
3.4
Mask R-CNN
HASSOD (Ours)
23.8
13.5
38.3
50.9
4.3
21.5
11.8
35.1
46.6
4.1
CutLER [38]
23.6
13.1
36.2
55.6
4,5
20.2
11.3
31.1
46.2
3.6
Cas. Mask R-CNN
HASSOD (Ours)
26.9
15.6
42.2
56.9
4.9
22.5
12.7
36.1
47.8
4.2
D Choosing Thresholds for Hierarchical Adaptive Clustering
When determining the merging thresholds {0....g) }. mainly consider our computational constraints and empirical observations. The decision of merging thresholds is not made based on validation performance (Table 2), ensuring that HASSOD is fully self-supervised
Number of pseudo-labels per image
100
80
60
40
20
0.0
0.1
0.2
0.3
0.4 0.5
0.6
0.7 0.8
Merging threshold
Figure 7: Relation between the number of pscudo-labels per image after post post-processing (y-axis) and the merging threshold gmerge (x-axis). When gmerge 2 0.5, the number of pseudo-labeled masks grows rapidly. Empirically, we find that when the number of masks per image exceeds 20, generating, loading, and transforming such pseudo-labels becomes major bottleneck in HASSOD. Therefore, we choose three thresholds (mmerge) = {0.1, 0.4), mainly guided by a computational consideration.
16

---

Guidance by number of pseudo-masks. Our choice for { is primarily guided by the number of pseudo-label masks produced per image. Figure shows the relationship between the number of masks per image and different thresholds. When Omerge 2 0.5, the number of masks per image escalates rapidly. This steep increase incurs significant computational costs, both during the initial generation of pseudo-labed and the subsequent data loading and pre-processing procedures during model training. To strike a balance between computational efficiency and the desired mask granularity, thresholds of {0,teree } {0.1, 0.2, 0.4) are chosen.
Table Generalizability of the merging thresholds (0 j. With the same merging threshold, the number of generated pseudo-labels is not significantly changing across different image datasets.
Number of pseudo-labels per image
mmerge
MS-COCO [20] Objects365 [27]
SA-1B [18]
0.1
2.58
3.49
2.91
0.2
4.20
5.78
4.88
0.4
11.61
12.15
12.70
Threshold generalization across datasets, Another noteworthy observation is the generalizability of these thresholds across various datasets. In Table 6, present the number of generated pseudo-labelt per image on three datasets. With the merging threshold (merge fixed, the number of generated labels relatively stable, regardless of the source image dataset. Therefore, our pre-set thresholds generalizable and require no further tuning when transferred to other datasets. Meanwhile, our detection model was trained MS-COCO images with pseudo-labels generated using our pre-set (0merss) }. and could generalize well to other datasets in a zero-shot manner, as shown in Table 1. This fact shows that the thresholds { are effective regardless of evaluation datasets.
E Computational Costs in Hierarchical Adaptive Clustering
We adopt the hierarchical adaptive clustering strategy to generate our initial pseudo-labels. this section, we provide additional details regarding computation costs in this procedure.
As shown Figure 3, the hierarchical adaptive clustering contains four steps "merge." "ensemble," " and "split." Among them, the "merge" step accounts for the major computation costs. We can analyze time complexity: Suppose we have n patches in the beginning. then there are at most n merging steps before stopping. Each merging step requires retrieving the most similar pair of adjacent regions. The collection of adjacent pairs has size at most (nn), and each operation requires time O(log n.) if this collection is organized as a balanced binary tree. Therefore, the merging process has time complexity (n l)g In our practice, the input image has resolution 480 x 480, so 480
x
480
3, 600. This time complexity is affordable.
More concretely, we list the time costs the hierarchical adaptive clustering on our computation platform in Table 7. Since the procedure is learning-fre free, we can parallelize the processing of images using more than one worker. On our computation nodes equipped with NVIDIA A100 GPUs, we can reduce the processing time to 1 1.59 sec/image. With 4 such nodes, we can complete the pseudo-label generation for MS-COCO train and unlabeled splits (0.24 million images) in 1. ~ day. 4x 86400
Table 7: Time costs in the steps of the hierarchical adaptive clustering. Notably, with multiple parallel workers, we can reduee the total processing time of MS-COCO [20] images to day.
Time Cost (sec/image)
Parallelized Cost (sec/image)
Step
Workers
Merge and Post-Process Ensemble and Split
11.7
8
1.46
2.1
16
0.13
Total
13.8
1.59
17

---

Impact of Patch Size in Hierarchical Adaptive Clustering
We use the DINO [5] pre-trained ViT-B/8 backbone to generate the initial pseudo-labels through our hierarchical adaptive clustering. We observe that the small patch size 8 x leads to better pseudo-label quality. In Table 8, we compare the mask quality of pseudo-labels generated by ViT-B/8 (patch size 8 8) vs. ViT-B/16 (patch size 16 x 16). For fair comparison, we use 480 480 input resolution for ViT-B/8 and 960 960 for ViT-B/16, so that they have the same number of initial patches. With the same merging threshold (merge ViT-B/16 leads to slightly fewer labels per image, and the quality is significantly worse than ViT-B/8, especially for small and medium objects. Therefore, we apply VIT-B/8 in our experiments for its localized visual features and subsequent high-quality pseudo-labels.
Table 8: Comparison between DINO ViT backbones with different patch sizes 8 x 8 and 16 x 16. The backbone with the smaller patch size leads higher-quality initial pseudo-labele the hierarchical adaptive clustering procedure.
DINO
Labels per Img
Mask ARM
gmerge
Backbone
AR
ARg
AR
AP
0.1
2.58
4.1
0.6
5.1
15.6
1.8
VIT-B/8
0.2
4.20
5.3
0.9
7.0
18.4
1.8
0.4
11.61
7.8
1.7
12.1
22.1
1.3
0.1
1.97
3.0
0.3
2.3
14.6
1.1
VIT-B/16
0.2
3.19
3,8
0.4
3.3
17.4
1.2
0.4
10.15
5.7
0.9
6.6
21.7
1.3
G Different Behaviors of SAM and HASSOD in Object Part Detection
Segment Anything Model (SAM) [18], supervised segmentation model, has demonstrated its ability creating high-quality, fine-grained segmentation of images. Directly comparing the performance between SAM and HASSOD would be unbalanced, considering SAM requires extensive human- labeled images for training, while HASSOD operates entirely under self-supervision. Despite this, remains intriguing to understand their different behaviors. In this section, we delve into qualitative comparison between SAM and our proposed HASSOD approach, paying particular attention their respective abilities to detect constituent parts of whole objects. Figure 8 presents visualization for this comparison.
Both SAM and HASSOD can perform fine-grained segmentation within complex scenes, successfully detecting individual object parts that constitute a whole entity. However, a clear distinction arises in their respective approaches towards the segmentation of these object parts. For scenes in which objects follow a grid pattern, where the whole entity is partitioned by regular boundary lines into semantically similar pieces, SAM tends to perceive each grid as distinct object. Conversely, HASSOD adheres to a holistic perspective for such cases, grouping the grid parts together even at the subpart level, due to their semantically analogous features.
In particular, HASSOD excels distinguishing object parts that contain different contents within whole objects. This skill of HASSOD is especially advantageous in certain real-world applications. For example, in medical imaging, HASSOD's fine-grained segmentation can potentially assist in identifying and separating different tissues or anatomical structures within scan. Additionally, in the manufacturing industry, HASSOD could be used in quality control to check and compare individual components of an assembled product. While both segmentation strategies of SAM and HASSOD are valid, HASSOD's unique ability to distinguish semantically different parts within objects provides distinct advantages in wide range of practical scenarios. Comprehensively quantifying and analyzing such an ability crucial for advancement of self-supervised object detection and segmentation approaches, and we consider it important future direction.
18

---

Input
SAM
HASSOD (Ours)
HASSOD (Ours) Whole Part Subpart
Whole Objects
Figure 8: Analysis different behaviors in object part detection between supervised SAM [18] and self-supervised HASSOD through qualitative visualization. While both models generate fine-grained segmentation, they exhibit distinct preferences concerning object parts. SAM inclines towards separating objects following grid pattern (e.g., comforters, keyboards, tiles), whereas HASSOD discriminates objects into semantically diverse parts (e.g., eyes and beard of a cat, handle and button of cabinet).
H Additional Evaluation Results for Completeness
As an addition to Table in the main paper, we present a comprehensive evaluation of HASSOD on the Objects365 [27], LVIS [11], and SA-IB [18] datasets Table 9. For a robust measure of performance, we incorporate the standard deviation, estimated from three independently trained models.
Table 9: Comprehensive evaluation results of HASSOD across three datasets. We set the maximum number of predictions per image 1,000. The number superscript on AR (e.g.. AR!0) denotes the number of most confident predictions taken into account when computing AR. The subscript on AP (e.g., APso) represents the Intersection-over-Urre predictions with ground truth labels. Size-specific metrics for small, medium, and large objects are indicated by the subscripts M. and μ, respectively. We include the standard deviation, estimated from three independent runs.
(loU) threshold utilized when matching
Dutaset
1 A0
AR0000
ARg
AR
ARE
AP
AP30
APTs
APs
APM
AP1
Box (Object
Detection)
15.20
36.63
39.03
21.40
40.43
$2.10
10.97
20.33
10.27
2.90
10.37
19.47
Objects365
+0.10
-0.06
+0.06
+0.10
+0.12
=0.17
+0.15
=0.31
+0.15
+0.10
=0.25
=0.25
15.64
56.90
4,94
2.82
LVIS [11]
10.07
20.02
10003
10.05
10.08
20.17
10.09
10.09
14.110
10.01
20.14
10.20
5.52
23.92
29.02
13.34
25.12
43.79
15.47
26,20
15,90
5.39
15.06
21.81
SA-1B [18]
1:0.02
20.03
20.18
10.27
10021
20.12
1008
20.09
1:0.03
11006
20.13
10.08
Mask
(Instance
Segmentation)
9.69
21.11
22.50
12.68
36.14
47.79
4.21
7.99
3.95
1.88
6.96
13.67
LVIS [11]
+:0.6
=0.05
+0.01
+0.01
+0.11
=0.17
+0.20
+0.25
+0.24
+0.19
=0.21
=0.18
5.27
21.62
25.99
12.90
22.76
38.33
13.85
24.78
13.67
4.09
13.45
20.36
SA-IB [18]
+-0.01
-0.07
+0.22
+-037
+0.24
-0.21
+0.09
+0.08
+0.11
+0.04
-0.10
19

---

I Limitations and Broader Impacts
Limitations. Due to the self-supervised nature of HASSOD, the learned object hierarchical levels may not be perfectly aligned with human perception. This mismatch may lead to over-segment or under-segment of objects in real-world applications.
7
7
Seunrins- briicke
Severing-
Koelnmesse
7
KoeInmesse
Koelnmesse
P
Im Sionstal
Im Sionstal
>
lm Sinnstal
Input
CutLER
HASSOD (Ours)
Overlapping and similar objects are hard distinguish. Middle: The whole object comprising diverse parts is not identified. Bottom: Text contents are not precisely localized.
Figure 9: Failure cases HASSOD, Top:
We observe that HASSOD performs unsatisfactorily in certain scenes due to this lack of human supervision. Figure 9 provides visualization of the failure cases. In the first row, HASSOD mistakenly treats the lower parts the two players single coherent object. Due their similar color and texture, multiple overlapping instances of the same class can sometimes be perceived as one object. In the second row, HASSOD fails predict a mask that encompasses the entire motorcycle. This object consists of highly heterogeneous parts, making it challenging to recognize them as components of a single entity. the third row, HASSOD fails to detect the text "Koelnmesse," " and the boundaries for other text are not clear. We also observe that such errors are not unique to HASSOD; they appear in prior self-supervised object detection methods like CutLER as well. We believe that further human supervision would be necessary for correcting these mistakes.
Broader impacts. Detecting object parts with self-supervision may be beneficial real-world applications including robotic manipulation and inspection. As general object detection method, we share risks associated with applying recognition models such as abuse of surveillance systems.
Hyper-Parameters and Implementation Details
In the initial pseudo-label generation process, we use a frozen ViT-B/8 model [8] pre-trained on unlabeled ImageNet [7] by DINO [5], self-supervised representation learning method, extract visual features of train and unlabeled images in MS-COCO [20]. Following prior work Cut- LER [38], we resize the resolution of each image to 480 x 480, leading to x 60 patches as initial regions. The merging process stops at three thresholds @ 0.4, ginere 0.2, ल = 0.1, and results from these three thresholds are ensembled after post-processing. The post-processing steps include Conditional Random Field (CRF) [19] and filling the holes in each mask. We also filter out low-quality masks that 1) have an antersetiionover-UUne
(IoU) smaller than 0.5 before and after CRF, 2) are smaller than 100 pixels, or 3) contain more than two corners of the image (which are likely
20

---

background). These post-proct proccsocssinng steps are also used in prior work including CutLER [38]. After ensembling results from the three thresholds { J==1. we identify the hierarchical levels of each object mask based on the coverage relation analysis. The coverage threshold is set to Beover = 90%.
In the detector training stage, train Cascade Mask R-CNN [4] with ResNet-50 [13] backhone on the same MS-COCO [20] images. The backbone is initialized from DINO [5] self-supervised pre-training. Our hyper-parameter setting for Mean Teacher mostly follows the practice of Unbiased Teacher [22]. The whole training process starts with a "burn-in" stage, during which the student model is only trained on the initial pseudo-labels with fixed learning rate 0.01 and fixed loss weights. After the burm-in stage, the teacher model is introduced, and we gradually adjust the learning rate from 0.01 the loss weight in the label-to-studet branch from 1.0 0.0, and the loss weight in the tottbrtt--tuuent branch from 2.0 to 3.0, all following a cosine schedule. The whole training process spans 40,000 iterations with a batch size of 16 images. The training performed on 4x NVIDIA A100 GPUs. Our code is developed based on PyTorch [24] and Detectron2 [40].
K Additional Qualitative Results
In this section, we present visualization detection results by CutLER [38] and HASSOD on Objects365 [27] in Figure 10 and SA-1B [18] in Figure 11 (see next pages). Qualitative results LVIS has been included in the main paper, 5.
21

---

M
01
M
Soney Hert
Sonry
Sumty
Sonny Mort
Input
CutLER
HASSOD (Ours)
HASSOD (Ours)
Whole Objects
Whole
Part
Subpart
Figure 10: Qualitative results on Objects365 [27]. In each row, we show detection results of prior state-of-the-art self-supervised object detector CutLER, whole objects predicted by HASSOD, and all object predicted by HASSOD.
22

---

INRAND
Input
CutLER
HASSOD (Ours)
HASSOD (Ours) Subpart
Whole Objects
Whole Part
Figure 11: Qualitative results on SA-1B [18]. In each row, we show detection results of prior -tate-of-the--rt self-supervised object detector CutLER, whole objects predicted by HASSOD, and all object predicted by HASSOD.
23


Reviews:
[{"review_id": "lH6sCdwSPo", "reviewer": "Reviewer_s64Y", "summary": "This paper proposes a hierarchical Adaptive Self-Supervised Object Detection (HASSOD), an approach that  learns to detect objects and understand their compositions without human supervision. \nHASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image.\nHASSOD identifies the hierarchical levels of objects in terms of compositionality, by analyzing coverage\nrelations between masks and constructing tree structures.\nThe proposed method adapts the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process.\n\nThe Mask AR is improved from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B.\n\n", "questions": "If this proposed work is inline with self-supervision,\nwhat are the equivalent/compatible stages of  - \npretext learning using base data with no labels;\nfine-tuning with support set;\ntarget set for inference on downstream task?\n\nA table specifying the dataset distributions (although names given) and stages exploiting the same\nfor self-supervision should have been given.\n\nAny scope of few-shot learning or meta-learning framework in your proposed approach?\n ", "limitations": "Lack of analytics ;\nNO ablation studies with cost functions;\nLack of mapping to standard Self-supervision paradigm, is a concern.\n\nObject Detection in complex cases of overlap of objects, background clutter not provided.\n\nI think its high time that researchers also look at identification/detection in camouflage (a personal opinion).\n\n\n\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Good results of the process, exhibited  in Fig. 3.\n\nMany quantitative comparisons in Tables 1-3.\n\nNice to read the overall paper.\n\nGood to see the reference of the vital work of [7], even after more than a decade.\n\nAppendix Doc highlights the importance of a better metric than AR.\n", "weaknesses": "\nLot of experiments, results, but no analytics. Not even a single equation, cost function, hyper-parameter set mentioned. \n\nAdaptation of mean-teacher model, still makes it a self-supervised framework.,\n\nThe overall designed architecture is explained with examples of images, in a diagram in fig. 2.\nA flowchart would still be better.\nWhich part requires CNN-based learning, which part requires shallow learning (if any) or heuristics is\nnot so clear (except a bit in Sec. 4.1, a small 2nd para).\n\nOnly in the Appendix - \nThe platform/environment/resources used for implementation are specified.\nThe computational time required for each sub-stage of the process is also not mentioned,\nincluding the inference time. This is very important for practitioners.\n\nFew failure cases should have been highlighted with causes/reasons; Appendix is not thorough on this too.\n\nColor labels used for inference vs bounding box, often causes confusion - the color overlap for anyone to clearly understand.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement of our “good results”, “quantitative comparisons”, and overall writing. We provide the following clarifications in response to your concerns:\n\n1. Lack of Analytics and Ablation Study Regarding Cost Functions and Hyper-parameters\n\n- Our central contribution is a **new learning paradigm**, rather than innovating on cost functions. We adopted a *minimalistic* design, using standard cost functions rooted in established literature. Most loss functions and hyper-parameters are inherited from previous works.\n\n-  To demonstrate the standard practices in our design, we can look into the two-stage learning of HASSOD (see more details in the general response):\n    - Stage 1: We apply the hierarchical adaptive clustering strategy, leveraging the DINO representations to derive initial pseudo-labels. The similarity measure in this process, as described in Line 118, involves computing the *pairwise cosine similarity* between adjacent region features: $\\frac{x^T y}{\\|\\|x\\|\\|_2\\|\\|y\\|\\|_2}$. This choice of metric is well-grounded in the self-supervised learning literature such as SimCLR and MoCo.\n   - Stage 2: This stage focuses on training our object detection and instance segmentation model Cascade Mask R-CNN. We optimize *conventional detection and segmentation losses*, including 1) the foreground/background classification loss, 2) bounding-box regression loss, and 3) mask prediction loss, following Mask R-CNN and Cascade R-CNN.\n\n- Presentation choices: In our initial submission, we consciously chose to minimize the use of equations, especially when the textual description was sufficient. Our goal was to maintain the paper's readability and prevent readers from being overwhelmed by equations. However, in light of your feedback, we will revise our paper with a more explicit introduction to the computational procedures, backed by relevant equations and analytics.\n\n- Ablation study clarification: Since most of our chosen loss functions and hyper-parameters are rooted in standard practice, we found it unnecessary to ablate these specific choices. Meanwhile, we had ablation studies for **our novel designs**: hierarchical level prediction (Section 3.2) and mean teacher training with an adaptive target (Section 3.3). The details are in Table 3.\n\n2. Self-Supervised Nature with Mean Teacher\n\nWe designed our method to be fully self-supervised relying on no labeled data, which we perceive as *a primary advantage*. We would be grateful for additional insights if there are specific concerns about our self-supervised strategy.\n\n3. Figure 2 and Clarification on Components Involving CNN Learning\n\nWe have revised this figure and included it as Figure R1 in the rebuttal PDF. As mentioned in the general response, HASSOD is a two-stage *discover-and-learn* approach. CNN-based learning happens in the second stage.\n\n4. Computational Details\n\nPlease check our general response.\n\n5. Lack of Failure Case Analysis\n\nWe appreciate your suggestion and have included some failure case analysis in Figure R4 in the rebuttal PDF.\n\n6. Overlap of Colored Masks and Bounding Boxes During Visualization\n\nWe followed standard visualization practices for instance segmentation as Mask R-CNN. The perceived clutter in our figures stemmed from depicting challenging cases with multiple objects in a scene. Our hierarchical predictions also resulted in overlapped masks. Based on your suggestion, we will omit colored bounding boxes and only show segmentation masks for better visual clarity. As an example, please check the failure case visualization mentioned above.\n\n7. Alignment with Standard Stages of Self-Supervision Paradigm\n\n- First, it is essential to note that our work does not delve into self-supervised **representation** learning, which is merely a subset of the broader self-supervised learning domain. Our primary focus is on self-supervised **object detection**, an inherently significant task in its own right. This task has a wide range of potential applications such as robotic vision. When contextualizing our work within the existing literature, our method aligns with the *discover-and-learn* paradigm by prior work such as FreeSOLO and CutLER.\n\n- The goal of our self-supervised learning task sets it apart from representation learning methods like SimCLR or MoCo. However, for the sake of the best understanding, the training of our model using the pseudo-labels produced by our clustering can be considered as the “pre-text” task, though the task goals are not generated on-the-fly. Moreover, our work does not involve any “fine-tuning” or “linear probing” on ground-truth labels, as our model intrinsically produces object-level bounding boxes and masks. Regarding “downstream evaluation”, we assess our model's performance on unseen datasets, like SA-1B, in a zero-shot setting.\n\n8. Dataset Specifics for Self-Supervision\n\nThroughout our self-supervised learning process, we solely used the `train` and `unlabeled` splits of MS-COCO, which contains 0.24 million images. This was detailed in Section 4.1.\n\n9. Incorporation of Few-Shot Learning or Meta-Learning\n\nWhile few-shot and meta-learning are valuable, our current research centers on fully self-supervised object detection. Integrating these techniques would be an interesting future exploration.\n\n10. Object Detection in Complex Scenarios\n\nIndeed, our model has been tested in challenging cases with object overlap and cluttered backgrounds. Please refer to our supplementary material, particularly Figure 6 (first row) and Figure 7 (last two rows). These illustrations demonstrate the advantage of our model over CutLER in complex scenarios."}, {"review_id": "fa6X1ix3Dd", "reviewer": "Reviewer_p4z2", "summary": "This goal of this paper is to improve the performance of self-supervised object detection by enhancing the detection pseudolabels. In addition to semantic masks provided as detection labels to an object detector, the authors utilize a rule-based approach to automatically generate a hierarchical structure of objects and provide it as targets to the detector. By training on hierarchical pseudolabels, the detector learns about the compositional structure of objects and therefore performs better. Additionally, the authors use a teacher-student network to train the detector instead of multi-round self-training approach. The proposed approach outperforms two other self-supervised detection approaches.", "questions": "- When using a single threshold to generate the masks, my understanding is that we should only get one level of objects. In other words, the output masks should not intersect with any other masks. How are the numbers in Table 2 generated when using a single $\\theta^{merge}$ threshold? This approach should only be possible when having intersecting masks (i.e., ensemble).", "limitations": "- Limitations about not aligning hierarchical levels with human perception are addressed. But it's missing limitations concerning the inability of this model to generate representations useful to classification of the detected boxes/masks.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors tackle an important problem in perception; the hierarchical structure of objects. By providing object labels at multiple levels in a tree structure, the detector can learn patterns in compositional structures of objects (subparts - parts - whole), which helps improve the predictions. For example, a model can learn to condition a whole object prediction on the existence of its parts and subparts.\n- This hierarchical structure is not only found in vision, but also other perceptual modalities and NLP. Therefore, understanding how to generate and utilize this structure can be helpful in other tasks and across other modalities.\n- This method is completely self-supervised, based on features extracted from DINO trained on ImageNet. The improvement in detection performance comes at no supervision cost, only additional computational cost to generate the pseudolabels.\n- The approach shows promising quantitative results, surpassing other self-supervised methods. ", "weaknesses": "- This approach, like many others, will always be upper bounded by the quality of the features and the robustness of the self-supervised representation learning approach used to generate them. The generation of labels depends on frozen features, so the detector will be sensitive to the quality of those features.\n- Approaches that use hierarchical clustering lose representation quality as they naively average high dimension feature representations to merge clusters. Therefore these methods cannot cluster the object representations to generate class labels for training. This is similar to approaches like TW-FINCH [1] that takes the same route for event segmentation. This being said, I'm interested in seeing whether high-level representations of objects generated by averaging low-level pixel/patch representation would provide good matching to true class labels. An alternative would be to add a (non-)linear on the average-based representations to classify the objects.  \n- While there is no supervision cost to generate the extra hierarchical labels, there is computational cost in clustering and structure discovery. It is important to share the computational cost in order to determine whether this approach is practical and can be scaled up.\n- I am not convinced that simple thresholding of the similarity metric provides accurate separation between the hierarchy levels. Using an arbitrary threshold will guarantee a tree structure but not necessarily a part-whole structure. The goal is not to find any tree structure whith parent masks covering children masks. These rules do not guarantee a part-whole structure. Each entity in the tree should be a coherent object (part-whole tree).\n- One suggestion is to plot $\\theta^{merge}$ against the number of objects in the image and detect regions with lowest slopes to be the thresholds. The intuition is that in these regions the model is less sensitive to changes in $\\theta$, therefore there could be good separation between hierarchical levels at these thresholds.\n- How do the authors decide on $\\theta^{merge}$? Is it using the ablations in Table 2? Table 2 is generated using groundtruth masks. So if the decision on $\\theta^{merge}$ is based on evaluating the masks with ground truth masks, this would be a significant flaw in the experimental protocol. It would violate the self-supervision claim. Usually researchers decide based on a small validation set, and even then it is still arguably supervised. This is an important point.\n\n[1] Sarfraz, Saquib, et al. \"Temporally-weighted hierarchical clustering for unsupervised action segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement that our method tackles “an important problem of hierarchical structure of objects”, is “completely self-supervised”, and shows “promising quantitative results”. We provide the following clarifications in response to your concerns:\n\n1. Upper Bound of Self-supervised Representation\n\nHASSOD is **not upper bounded** by such representations.\n\n- Starting from existing self-supervised representations is a standard practice in the literature of self-supervised object detection. For instance, FreeSOLO utilizes pre-trained DenseCL representations, and the state-of-the-art method CutLER relies on DINO representations. Pre-trained self-supervised representations have been shown as a good starting point for self-supervised object detection. We are also following this paradigm.\n\n- The performance of our method is **not upper bounded** by the constraints of the frozen self-supervised models. Two key insights in HASSOD help us break through the limits of DINO features by self-improvement: 1) We train a detection model to learn common objects and their hierarchical relations for *enhanced generalization* to unseen images. 2) We leverage Mean Teacher for continual self-enhancement, and gradually *minimize the negative impact of noisy initial pseudo-labels*.\n\n- A close examination of Table 2 and Table 3 in our paper demonstrates that HASSOD is not upper bounded by DINO representations. Specifically, the quality of pseudo-labels generated by frozen DINO features can be quantified by the best average recall (AR) and average precision (AP) listed in Table 2 as 8.9 and 1.8, respectively. After training a detection model with HASSOD, these values are boosted to 22.4 and 6.3, respectively, as shown in Table 3. \n\n2. Object Representations for Classification\n\n- Firstly, our primary task is **class-agnostic** self-supervised object detection, established by prior work such as FreeSOLO and CutLER. Distinct from the traditional class-aware object detection tasks, this direction is motivated by numerous scenarios where the primary goal is not to recognize specific object classes but rather to capture objects within an image. As examples in robotic vision, a robot may need to identify and navigate around obstacles without classifying them.\n\n- Meanwhile, we acknowledge the value in examining our extracted features for class-aware object detection. To this end, we tune a set of classification heads newly attached to the pre-trained Cascade Mask R-CNN model. The other modules are initialized from CutLER or HASSOD and frozen during the supervised fine-tuning on MS-COCO. The results of this classification-only tuning are shown in the table below. The model pre-trained by HASSOD achieves $2\\times$ AR and AP in class-aware object detection and instance segmentation, suggesting that HASSOD has gained representations more helpful for classification during the self-supervised learning procedure, as compared with CutLER.\n\n|Initialization|AR|AR$_S$|AR$_M$|AR$_L$|AP|\n|---|---|---|---|---|---|\n|CutLER|4.2|1.1|2.8|6.3|1.5|\n|HASSOD (Ours)|**8.4**|**1.6**|**6.8**|**14.2**|**3.1**|\n\n3. Computational Costs\n\nPlease check our general response.\n\n4. Simple Thresholding for Accurate Hierarchy Levels\n\nYour concerns over the use of thresholding for deriving hierarchical structures are valid. Indeed, using simple thresholding could potentially lead to deviations from the expected part-whole structure that we typically associate with objects. For instance, there might be instances where we generate a structure that is closer to an incomplete whole object rather than a distinct part.\n\nDespite this concern, thresholding in conjunction with the DINO features leads to surprisingly coherent and semantically meaningful results. Here are our observations:\n\n- Coherence of pseudo-labels: From our extensive experiments and observations, the pseudo-labels generated are **predominantly coherent**, displaying uniformity in color and edge characteristics. We rarely find significant heterogeneity in the pseudo-labels. In Figure 2, we showed a real example from our pseudo-labels. This essentially implies that the hierarchical structures produced by thresholding and DINO are *surprisingly consistent* with the part-whole structures that we, as humans, commonly perceive.\n\n- Robustness of DINO Features: The quality and efficiency of the hierarchical structures are highly attributed to the DINO features. Prior work [Ref1, 2, 3] has demonstrated DINO's ability in identifying **whole objects**. Impressively, DINO features are robust with respect to various types of thresholds. Our contribution is to extend this understanding further by not only segmenting whole objects but also extracting **meaningful and coherent parts of objects**.\n\n5. Deciding and Plotting Thresholds $\\theta^\\text{merge}$\n\nPlease check our general response.\n\n6. Table 2 with Single $\\theta^\\text{merge}$ Threshold\n\nIn Table 2, for the rows with one single $\\theta^\\text{merge}$ threshold, as the reviewer correctly pointed out, we would produce a collection of pseudo-labels, in which each mask does not intersect with others. However, Table 2 aims to evaluate the quality of these generated *initial pseudo-labels* as individual entities. The evaluation does not involve the hierarchical relationships between the labels. Instead, it provides an assessment of how well these pseudo-labels align with true object masks in the image based on the selected threshold. The hierarchical structure of masks is learned and assessed in the subsequent stage of HASSOD.\n\n[Ref1] Siméoni et al. Unsupervised Object Localization: Observing the Background to Discover Objects. In CVPR, 2023.\n[Ref2] Wang et al. Cut and Learn for Unsupervised Object Detection and Instance Segmentation. In CVPR, 2023.\n[Ref3] Hamilton et al. Unsupervised Semantic Segmentation by Distilling Feature Correspondences. In ICLR, 2022.\n"}, {"review_id": "od4NhSvQq0", "reviewer": "Reviewer_cMyQ", "summary": "The authors propose a self-supervised object detection approach by employing a self-supervised pre-trained model (i.e., DINO) to hierarchically and adaptively group regions into object masks using multiple pre-defined thresholds based on cosine similarity in the feature space. The authors then adapt the Mean Teacher framework to train a student object detector with the initial pseudo-labels from the clustering process as well as the progressively refined pseudo-labels from the teacher model. Extensive experiments demonstrate the superiority of the proposed approach over existing self-supervised object detection and instance segmentation methods.", "questions": "See questions mentioned above. I am currently leaning towards borderline reject and hope the authors could address my concerns during the rebuttal.", "limitations": "The authors have well addressed the limitations and the broader impacts in Sec. 5, which looks good to me.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tThe experiments are extensive and the results are promising.\n-\tThe paper is generally well-written and easy to follow.", "weaknesses": "-\tThe authors use a set of pre-defined thresholds (i.e., {0.1, 0.2, 0.4}) to merge region features. However, considering different datasets may have different distributions, such COCO-tuned hyper-parameters may not be suitable when merging region features for other datasets. Thus, the authors have to select the optimal thresholds for each dataset when performing clustering, which limits the practical applications of the proposed method to some extent.\n-\tThe authors use some post-processing techniques like CRF to refine the masks. I am curious about the mask quality without such post-processing. Will the performance be dropped significantly?\n-\tAlthough the proposed method can segment an image with hierarchy, it seems that the mask quality is somewhat inferior to CutLER according to the qualitative results as shown in the paper.\n-\tThe authors choose DINO with ViT-B/8 model, i.e., each spatial element in the feature map corresponds to a 8x8 patch in the original image. What is the effect of the patch size? It seems that the patch size tends to affect the performance significantly and should also be ablated.\n-\tThe authors use Cascade Mask R-CNN to verify the effeteness of the proposed method. What about other kinds of detectors? It would be better to provide results based on more detectors to verify the generality of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement that our “experiments are extensive”, “results are promising”, and “paper is well-written and easy to follow”. We provide the following clarifications in response to your concerns:\n\n1. Generalizability of Pre-Defined Thresholds\n\nPlease check our general response.\n\n2. Mask Quality Without CRF\n\nThe use of the CRF post-processing is necessary for acquiring precise edges in pseudo-labeled masks. When generating initial pseudo-labels, we employ ViT-B/8 to extract features at the patch-level, with each patch being of the size 8x8. Absent CRF post-processing, the masks would inherently be composed of these 8x8 patches. As a consequence, the boundaries of such masks would resemble a jigsaw puzzle and would not be consistent with the real boundaries of objects. Meanwhile, we adopt CRF post-processing following the foundational segmentation work DeepLab [Ref1], as well as prior methods on self-supervised object detection including TokenCut and CutLER. For example, CRF post-processing has been shown necessary in Figure 4 of TokenCut [Ref2].\n\n3. Inferior Mask Quality\n\nIt is true that there are some instances where the quality of masks generated by our method may seem inferior when compared directly to CutLER. There are a couple of reasons for this:\n\n- Training duration: As indicated in Section 4.1, our model was trained for a considerably shorter duration compared to CutLER – specifically only **1/12** of CutLER's total iterations. This was mainly due to our computational resource constraints. We believe that with prolonged training, our model has the potential to produce higher quality masks.\n\n- Focus on overall performance: It is also crucial to highlight that our primary objective was to improve the overall detection and segmentation performance by increasing the number of detected objects, especially in challenging scenarios. While we also strive for mask perfection, our main focus is still on improving the overall average recall and precision. As evidenced in Table 1, our approach does demonstrate superior overall performance in instance segmentation, which we consider a significant achievement.\n\n4. Impact of Patch Size in DINO ViT-B/8\n\nWe appreciate your query regarding the ViT patch size and have produced the following table as additional ablation study. For fair comparison, we use 480x480 input resolution for ViT-B/8 and 960x960 for ViT-B/16 so that they have the same number of patches. With the same $\\theta^\\text{merge}$, ViT-B/16 leads to slightly fewer labels per image, but the quality is significantly worse than ViT-B/8, especially for small and medium objects. Therefore, we apply ViT-B/8 in our experiments for its localized visual features and subsequent high-quality pseudo-labels.\n\n| DINO Backbone | $\\theta^\\text{merge}$ | Labels per Image | AR | AR$_S$ | AR$_M$ | AR$_L$ | AP |\n|---|---|---|---|---|---|---|---|\n| ViT-B/8  | 0.1 | 2.58 | 4.1 | 0.6 | 5.1 | 15.6 | 1.8 |\n| ViT-B/16 | 0.1 | 1.97 | 3.0 | 0.3 | 2.3 | 14.6 | 1.1 |\n| ViT-B/8  | 0.2 | 4.20 | 5.3 | 0.9 | 7.0 | 18.4 | 1.8 |\n| ViT-B/16 | 0.2 | 3.19 | 3.8 | 0.4 | 3.3 | 17.4 | 1.2 |\n| ViT-B/8  | 0.4 | 11.61| 7.8 | 1.7 | 12.1| 22.1 | 1.3 |\n| ViT-B/16 | 0.4 | 10.15| 5.7 | 0.9 | 6.6 | 21.7 | 1.3 |\n\n5. Use of Cascade Mask R-CNN\n\n- We chose Cascade Mask R-CNN to ensure **fair comparison with CutLER**. The same detector architecture enables a fair and direct comparison between our proposed method and CutLER. Otherwise, it would be unclear whether the performance gain came from the improved learning paradigm or a stronger detector architecture.\n\n- We do acknowledge the importance of demonstrating the versatility of our method across different detectors. To this end, we are conducting experiments using our proposed method and Mask R-CNN, and we will share the results within the next week.\n\n[Ref1] Chen et al. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. In TPAMI, 2017.\n[Ref2] Wang et al. TokenCut: Self-supervised Transformers for Unsupervised Object Discovery using Normalized Cut. In CVPR, 2022.\n\n"}, {"review_id": "U0b15WREUf", "reviewer": "Reviewer_wiMX", "summary": "The paper proposes a self-supervised object detection method (in fact, it can also do instance segmentation). This is based on an idea of bottom-up merge by strong feature representation like from DINO feature. The image patch grouping is hierarchical because of setting different stopping thresholds. This makes the model interpretable. Overall, the idea is simple and straightforward. The model can be trained by teacher-student framework. Experiments show better results than recent work MaskCut.", "questions": "Some critical details are missing in the paper and they need to be clarified. 1) What is learned, the whole DINO backbone or just a head network? 2) For the Mean-Teacher, what are the losses used to supervise the student networks training? \n\nThe grouping process in section 3 is iterative and seems to be too slow. What is the time complexity?\n\nLine 138, how to ensemble?\n\nLine 195, in section 3, looks like only the feature representation is used, why train a Cascade mask rcnn here? Obviously, Cascade mask rcnn is not self-supervised.\n\nLine 233 and 234 are inconsistent with section 4.2. What exactly Objects365, Lvis, SAM are for evaluation or training?\n\nLvis should have a LVIS-rare benchmark.\n\nWhole, part, subpart should be given rigorous definitions.\n\nFor the limitations 2) and 3) in the second paragraph, I don't understand how they are solved after reading the whole paper.", "limitations": "The discussion at the end of the paper is good.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- the bottom-up merging method to solve the self-supervised object detection might be new.\n- Figure 1 and 2 are good to understand the method.\n- experiment results look good.", "weaknesses": "- The writing is bad. Many critical details are omitted which makes it super hard to under the method.\n- Some inconsistencies in the paper, needs to clarify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement that our method is “new”, figures are “good to understand”, and “experiment results look good”. We provide the following clarifications in response to your concerns:\n\n1. Clarification on Learned Network and Mean Teacher\n\n- We learn the weights in both the DINO backbone and the detection head network. To be precise, our approach adopts a two-stage discover-and-learn procedure, and all weight learning happens in the second stage. Please see more details in the general response.\n\n- In Mean Teacher, the student network “receives supervision from two sources simultaneously” (Line 172): One source is the teacher’s predictions, and the other is the pseudo-labels generated in the first stage. The training losses are the *standard object detection losses*, including classification loss, bounding box regression loss, and mask prediction loss. Formally, the training loss for the student is $L_\\text{student}=\\alpha_\\text{teacher}L_\\text{det}(\\tilde y_\\text{student}, \\tilde y_\\text{teacher})+\\alpha_\\text{pseudo}L_\\text{det}(\\tilde y_\\text{student}, y_\\text{pseudo})$, where $\\tilde y_\\text{student}$ is the student's prediction for the given image, $\\tilde y_\\text{teacher}$ is the teacher's prediction, $y_\\text{pseudo}$ is the pseudo-label, and $L_\\text{det}$ is the standard detection loss mentioned above. $\\alpha_\\text{teacher}, \\alpha_\\text{pseudo}$ are loss weights dynamically adjusted during training, where $\\alpha_\\text{teacher}$ linearly increases while $\\alpha_\\text{pseudo}$ is linearly decreasing, as introduced in Line 183-186.\n\n2. Time Complexity for Grouping Process\n\nSuppose we have $n$ patches in the beginning, then the time complexity of the grouping process is $O(n\\log n)$: We have at most $n$ merging steps before stopping. Each merging step requires retrieving the most similar pair of adjacent regions. The collection of adjacent pairs has size at most $O(n)$, and each operation requires $O(\\log n)$ time if this collection is organized as a balanced binary tree. Therefore, the grouping process has time complexity $O(n\\log n)$. In our practice, the input image has resolution $480\\times 480$, so $n=60\\times 60=3600$. This time complexity is affordable. For more statistics, please check the general response.\n\n3. Clarification on Ensemble\n\nIn Line 138, “ensemble” means to aggregate all the pseudo-labels generated from different merging thresholds $\\theta_i^\\text{merge}$.\n\n4. Cascade Mask R-CNN Training\n\nAs explained above, our method takes a two-stage procedure. We train a Cascade Mask R-CNN in the second stage, using the pseudo-labels generated in the first stage. Since the pseudo-labels are generated from self-supervised features, our approach is still *fully self-supervised*.\n\n5. Ambiguity in Dataset Usage\n\nThank you for pointing out the ambiguity. We should parse the sentence as “We evaluate ... (Cascade Mask R-CNN trained by HASSOD) on Objects365, LVIS, and SA-1B”, rather than “We evaluate ... Cascade Mask R-CNN (trained by HASSOD on Objects365, LVIS, and SA-1B)”.\n\nTo clarify, the Cascade Mask R-CNN learned by HASSOD is exclusively trained on MS-COCO images. The sentence in question means that after training, we evaluate the performance of this model on the Objects365, LVIS, and SA-1B datasets in a *zero-shot* manner. The model is not further trained on these datasets.\n\n6. LVIS-rare Benchmark\n\nOur main objective is to identify **every object** present in scene images without distinguishing based on classes. We train and evaluate our model as a class-agnostic detector, aligning with the methodology of prior research like CutLER, as mentioned in Line 210-212 of our paper. Consequently, when evaluating on LVIS, it is necessary for us to consider **objects from all categories**, irrespective of their categories as rare, common, or frequent.\n\n7. Rigorous Definitions of Whole, Part, and Subpart\n\nWe will revise our paper based on these concepts:\n\n- Whole: The entirety of an object, encompassing all its parts and subparts. In our method, a “whole” object is the largest coherent grouping of pixels that the algorithm identifies as an entity. For instance, when considering an image of a bicycle, the “whole” object would refer to the entire bicycle, including all its parts such as wheels, frame, and handle.\n\n- Part: A smaller component of the “whole” object that has consistent features but does not encompass the entire object. In the bicycle example, the wheel or the handle would be considered a “part” of the bicycle.\n\n- Subpart: An even smaller component of a “part”. It provides finer granularity in object segmentation. In the bicycle example, if we consider the wheel as a “part”, then the spokes and tire of the wheel could be considered “subparts”.\n\n8. Overcoming Previous Limitations\n\n- Limitation 2 - Narrow object coverage: Many prior methods, including TokenCut and MaskCut from CutLER, can only detect a *pre-defined number of objects* in images due to their algorithm design. However, our hierarchical adaptive clustering (Section 3.1) can recognize a broader range of objects, with the *number adaptively based on image content*. This approach markedly improves object detection, surpassing prior methods. As shown in Table 1, we boost Average Recall (AR) for self-supervised methods on SA-1B from 17.0 to 26.0.\n\n- Limitation 3 - Inefficiency: Prior methods like FreeSOLO and CutLER involve *multiple rounds of self-training*, increasing computational demands. For the first time, we introduced the Mean Teacher paradigm to our framework (Section 3.3), which allows a teacher model *continually produce pseudo-labels*, with a student model *simultaneously learning*. Compared to traditional multi-round self-training, our approach is smooth and efficient. Our method takes only 1/12 of the training iterations (Line 205) to outperform CutLER."}]

Author rebuttal:
In this response, we provide clarification on the common questions and concerns raised by the reviewers.

1. Clarification on Learning Procedure (Reviewers s64Y, wiMX)

Overall, HASSOD adopts a two-stage *discover-and-learn* approach, as illustrated in Figure R2 in the rebuttal PDF. This two-stage approach is not only intuitive but also a standard practice in the self-supervised object detection domain, as evidenced by literature such as FreeSOLO and CutLER.

- Stage 1 - Initial pseudo-label discovery: We employ our hierarchical adaptive clustering strategy (Section 3.1) to derive initial pseudo-labels. This process is based on a *frozen* DINO model, and it does *not* require learning any parameters.

- Stage 2 - Object detector learning: We *train a Cascade Mask R-CNN model* using the pseudo-labels from the first stage, enhanced by our hierarchical level prediction (Section 3.2) and Mean Teacher self-training (Section 3.3). This model has learned to detect and segment objects, offering *enhanced generalization* to images it has not seen during training, since it can learn from common objects and their relations in different training images.

2. Computational Costs (Reviewers s64Y, p4z2)

We apologize for any oversight in clearly presenting the computational platform, resources, and processing times within the main body of the paper. While we did mention some details regarding the *training data and iterations* in Section 4.1 and provided an extended discussion on the *computational platform* in Section F of the supplementary material, we understand the necessity for more comprehensive information. To address this, we have prepared the following table regarding the computation costs in pseudo-label generation.

|Step|Time Cost (sec/image)|Workers|Parallelized Cost (sec/image)|
|---|---|---|---|
|Merge and Post-process|11.7|8|1.46|
|Ensemble and Split|2.1|16|0.13|
|Total|13.8|-|1.59|

We use both the `train` and `unlabeled` splits of MS-COCO, totaling to about 0.24 million images. On computation nodes equipped with $4\times$ NVIDIA A100 GPUs, we can parallelize the processing of images, reducing the time to 1.59 sec/image. With 4 such nodes, we can complete the pseudo-labels generation for 0.24 million images in $1.59\times0.24\times10^6/86400/4\approx1$ day. Our method can be readily extended to an even larger scale of data with more parallel compute nodes.

For model training, the Cascade Mask R-CNN training procedure takes about 20 hours on our node with $4\times$ NVIDIA A100 GPUs. During inference, each image takes 0.15 second on average.

We will include the computational information in our revision.

3. Deciding Thresholds $\theta^\text{merge}$ (Reviewers p4z2, cMyQ)

Instead of relying on the validation performance shown in Table 2, we chose the merging thresholds $\theta_i^\text{merge}$ purely based on computational considerations and empirical observations, ensuring that our approach is fully self-supervised.

- Guidance by number of pseudo-masks: Our choice for $\theta^\text{merge}$ was primarily guided by the **number of pseudo-label masks produced per image**. In the rebuttal PDF, Figure R3 shows the relationship between #masks/image and different thresholds. When $\theta^\text{merge} \ge 0.5$, the number of masks per image escalates rapidly. This steep increase incurs *significant computational costs*, both during the initial generation of pseudo-labels and the subsequent data loading and pre-processing procedures during model training. To strike a balance between computational efficiency and the desired mask granularity, thresholds of $\theta_i^\text{merge} \in \{0.1, 0.2, 0.4\}$ were chosen. This choice is consistent with the suggestion by Reviewer p4z2.

- Threshold generalization across datasets: Another noteworthy observation is the generalizability of these thresholds across various datasets. In the following table, we present the **number of generated pseudo-labels per image** on three datasets. With the merging threshold $\theta^\text{merge}$ fixed, the number of generated labels is relatively stable, regardless of the source image dataset. Therefore, our pre-set thresholds are generalizable and require no further tuning when transferred to other datasets. Meanwhile, our detection model was trained on MS-COCO images with pseudo-labels generated using our $\theta_i^\text{merge}$, and could generalize well to other datasets in a zero-shot manner, as shown in Table 1. This fact shows that the $\theta_i^\text{merge}$ are effective regardless of evaluation datasets.

|$\theta^\text{merge}$|MS-COCO|Objects365|SA-1B|
|---|---|---|---|
|0.1|2.58|3.49|2.91|
|0.2|4.20|5.78|4.88|
|0.4|11.61|12.15|12.70|



Meta-review: This paper addresses the problem of class-agnostic object detection under the self-supervised learning setting by leveraging DINO's capability of forming the hierarchical whole-part pseudo labels to train cascade Mask R-CNN using the mean-teacher pipeline. The paper has been reviewed by several knowledgeable reviewers. The majority consensus from the reviews was that the paper will be of interest to the community and should be accepted. The reviewers acknowledged the authors' rebuttal efforts very positively. This meta-review concurs, and recommends acceptance.

However, as noted by the reviewers, there are some issues that need to be addressed carefully in revision, e.g.,

Whether the performance of the proposed method will "be upper bounded by the quality of the features and the robustness of the self-supervised representation learning approach used to generate them" (p4z2). The authors and p4z2 had in-depth discussions, which should be carefully justified and integrated into the main paper in revision.

Lack of analytics ; NO ablation studies with cost functions (s64Y)

Clarification on learning procedure (s64Y and wiMX). The revised figures in the rebuttal should be presented in the main paper in revision with precise explanations.
Decision: Accept (poster)