SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Objects365 [27] is a large-scale object detection dataset containing 365 object categories. The combined validation sets of Objects365 v1 and v2 include 80,000 images in total.
- LVIS [11] is a dataset that features a wide variety of over 1,200 object classes, using the same images as MS-COCO [20]. LVIS v1.0 validation set has 19,809 images, each annotated with object masks for instance segmentation.
- SA-1B [18] is a recent dataset that includes 11 million images and 1 billion fine-grained, model-generated object masks. SA-1B provides a more comprehensive coverage of all potential objects, facilitating a more robust evaluation of self-supervised object detectors. As SA-1B does not provide a validation split, we utilize a random subset of 50,000 images for our assessment.
In terms of evaluation metrics, we focus primarily on average recall (AR) rather than average precision (AP). The choice of AR over AP is motivated by the nature of the self-supervised task. In a dataset with a fixed number of classes, objects not labeled by humans – simply because they do not fall under the designated classes – may still be detected by a self-supervised detection model. Standard AP calculation would penalize such predictions as false positives, despite them being valid detections. In contrast, AR does not suffer from this issue, making it a more appropriate metric for our context. By prioritizing recall, we can more accurately assess the ability of our model to identify all relevant objects in an image, which aligns with the goal of the self-supervised object detection task. We evaluate AR based on both bounding boxes for object detection (“Box AR”) and masks for instance segmentation (“Mask AR”). Appendix A discusses the evaluation metrics in detail.
4.3 Self-Supervised Detection and Segmentation
After we use HASSOD to train the object detection and instance segmentation model, Cascade Mask R-CNN, on MS-COCO images, we evaluate our model on Objects365, LVIS, and SA-1B datasets in a zero-shot manner, i.e., no further training on these three datasets. The whole training and evaluation process is repeated for three times, and we report the mean performance for conciseness. The standard deviation of  $AR$  is less than 0.6 on all three datasets. Complete evaluation results are included in Appendix H.
We compare HASSOD with prior state-of-the-art self-supervised object detection methods, including FreeSOLO [37] and CutLER [38]. We also include results from SAM [18], the latest supervised class-agnostic detection/segmentation model, to gain understanding of the gap between self-supervised and supervised models, and how HASSOD is effectively closing this gap. To be consistent with other models, we provide SAM with only the raw images but no bounding boxes or points as prompts. For prior methods FreeSOLO, CutLER, and SAM, we directly evaluate the publicly available model checkpoints on the given datasets. Considering that the number of ground-truth labels per image may be greater than 100, we allow all models to output up to 1,000 predictions per image.
As shown in the main results summarized in Table 1, HASSOD significantly improves the detection and segmentation performance over previous self-supervised models FreeSOLO and CutLER. On Objects365, we improve the Box AR by 3.2; on LVIS, we improve Box AR by 3.3, and Mask AR by 2.3. The most remarkable performance gain is observed on SA-1B. We improve Box AR from 18.8 to 29.0 (relatively +54%) and improve Mask AR from 17.0 to 26.0 (relatively +53%).
We gain improved recalls for objects of all scales, but small and medium-sized objects relatively benefit more than large objects. For instance, our  $\mathrm{AR}_S$  is  $2.6\times$  as CutLER's  $\mathrm{AR}_S$  on SA-1B. It is worth noting that detecting small objects is intrinsically harder than large objects - even though the
Table 1: Comparison of self-supervised object detection and instance segmentation methods on prevalent image datasets. We consider the average recall (AR) instead of average precision (AP) as the main metric, because valid detection of objects outside the categories defined by human annotations is penalized by AP. HASSOD significantly outperforms the previously best methods FreeSOLO [37] and CutLER [38] in terms of AR at all object scales (Small, Medium, and Large). To understand the extent of improvements, we also include results from state-of-the-art supervised model SAM [18]. HASSOD leads to a reduced gap between fully self-supervised models and supervised SAM. Notably, HASSOD only uses  $\frac{1}{5}$  of training images and  $\frac{1}{12}$  of training iterations as CutLER.
Method	Box	Mask
AR	\( AR_S \)	\( AR_M \)	\( AR_L \)	AP	AR	\( AR_S \)	\( AR_M \)	\( AR_L \)	AP
Objects365 [27]
SAM [18]	54.9	32.1	60.5	67.6	11.9
FreeSOLO [37]	10.2	0.2	5.8	23.4	3.4	No ground-truth mask annotations in Objects365
CutLER [38]	35.8	17.6	36.1	50.5	11.5
HASSOD (Ours)	39.0	21.4	40.4	52.1	11.0
LVIS [11]
SAM [18]	42.7	27.7	66.3	75.5	6.1	46.1	31.1	71.3	74.6	6.7
FreeSOLO [37]	6.4	0.3	9.7	34.6	1.9	5.9	0.2	9.2	31.7	1.9
CutLER [38]	23.6	13.1	36.2	55.6	4.5	20.2	11.3	31.1	46.2	3.6
HASSOD (Ours)	26.9	15.6	42.2	56.9	4.9	22.5	12.7	36.1	47.8	4.2
SA-1B [18]
SAM [18]	60.5	19.8	59.8	81.5	38.2	60.8	20.0	59.9	82.2	38.9
FreeSOLO [37]	2.4	0.0	0.1	7.4	1.5	2.2	0.0	0.2	6.9	1.5
CutLER [38]	18.8	5.1	14.6	32.8	9.0	17.0	4.9	13.9	28.5	7.8
HASSOD (Ours)	29.0	13.3	25.1	43.8	15.5	26.0	12.9	22.8	38.3	13.8
labels in SA-1B are produced by the same SAM model, when the bounding box prompts are no longer available, SAM can only reach a 20.0 Mask AR for small objects. Meanwhile, we halve the performance gap between self-supervised CutLER and supervised SAM from 15.1 Mask AR $_S$  to 7.1 Mask AR $_S$ . By learning from hierarchical levels of object compositions, HASSOD more effectively captures small objects which are part of whole objects.
4.4 Qualitative Results
In this section, we analyze some qualitative results on images from LVIS. The visualization is shown in Figure 5. Qualitative results on other datasets are included in Appendix K.
As shown in the examples, our proposed HASSOD exhibits a more comprehensive coverage of all objects in complex scenes, compared with the previous state-of-the-art self-supervised object detection method CutLER [38]. This advantage originates from the pseudo-labels generated by our hierarchical adaptive clustering, which includes a proper number of candidate objects per image according to the image contents, rather than only focusing on a fixed number of objects. Furthermore, HASSOD can predict the hierarchical level of each detected object. Being a fully self-supervised model, HASSOD has surprisingly gained the human-like ability to comprehend the composition of objects. This ability leads to better interpretability and controllability: Users of HASSOD can understand the composition of each object detected by the model. Meanwhile, users can also control the segmentation granularity by selecting the predictions at the desired hierarchical level.
The qualitative results also show some limitations of HASSOD. By comparing the last two columns, it can be observed that HASSOD produces relatively fewer predictions for "subpart" objects. This is due to the distribution imbalance in the initial pseudo-labels, in which only about  $10\%$  objects are subparts. Also, the hierarchical levels learned by HASSOD are sometimes inconsistent with human perception. For example, instead of recognizing the person as a whole object in the last example image, HASSOD detects the upper and lower parts of the body as whole objects. Due to the lack of human supervision, the hierarchical prediction of HASSOD is not always aligned with humans. We further analyze this limitation in Appendix I.
Figure 5: Qualitative results on LVIS images. Overall, our HASSOD successfully detects more objects compared with CutLER [38]. CutLER tends to detect only one or few prominent objects in the image, while HASSOD captures other objects as well (e.g., bread in row 1, and traffic sign in row 3). Moreover, HASSOD learns the composition of objects (e.g., cat-face-eye in row 2, and vehicle-wheel-tire in row 4), which is similar to human perception.
In this section, we conduct an ablation study to understand the effects of each component in HASSOD. To evaluate the performance more robustly against a larger set of human-annotated object-level labels, we combine the annotations of MS-COCO [20] and LVIS [11] on the va12017 split, because they are complementary to each other: LVIS uses the same images as MS-COCO, but labels more object classes. However, LVIS annotations are not as exhaustive as MS-COCO, meaning that objects within LVIS categories may not be labeled on all images. After combining the two sets of annotations and removing duplicates, there are around 20 object-level labels per image. We use this combined dataset for evaluation in all the ablation study experiments, unless otherwise specified.
Quality of initial pseudo-labels. We first examine how the design choices in our hierarchical adaptive clustering influence the quality of initial pseudo-labels. The results are summarized in Table 2. Each threshold  $\theta_{i}^{\mathrm{merge}}\in \{0.1,0.2,0.4\}$  leads to a different trade-off between the number of labels per image and the recall. A higher threshold stops the merging process earlier, and thus results in more pseudo-labels. With post-processing, we can improve pseudo-label quality by removing about half of the labels, and increase AP significantly without losing much AR. Finally, the ensemble of multiple merging thresholds  $\theta_{i}^{\mathrm{merge}}\in \{0.1,0.2,0.4\}$  brings the best overall pseudo-label quality. Appendices D, E, and F present more details regarding the choice of  $\theta^{\mathrm{merge}}$ , computation costs, and ViT backbones in this stage.
Effects of hierarchical level prediction and Mean Teacher. After generating the initial pseudolabels with hierarchical adaptive clustering, we train the object detector with several techniques, including hierarchical level prediction, Mean Teacher self-training, and adaptively adjusting learning targets. The contribution of each technique is summarized in Table 3. Each component brings an additional 0.3 - 0.5 Mask AR improvement, and when they function together, the best overall performance can be achieved.
Table 2: Ablation study on factors influencing the quality of initial pseudo-labels. The threshold  $\theta^{\mathrm{merge}}$  controls the stopping criterion of the merging process, and affects AR and the number of pseudo-labels. Applying post-processing can remove low-quality pseudo-labels without decreasing AR by a large margin. Ensemble of multiple pseudo-label sources leads to the best overall quality.
Method	θmerge	Post-Process	Labels per Img	AR	ARS	Mask ARM	ARL	AP
MaskCut [38]	-	✓	1.85	3.5	0.0	2.0	20.0	1.5
Hierarchical adaptive clustering (Ours)	0.1	5.33	4.4	0.8	5.1	16.6	1.2
0.1	✓	2.58	4.1	0.6	5.1	15.6	1.8
0.2	8.36	5.5	1.2	7.0	18.9	1.3
0.2	✓	4.20	5.3	0.9	7.0	18.4	1.8
0.4	23.33	7.9	2.1	12.0	21.7	0.7
0.4	✓	11.61	7.8	1.7	12.1	22.1	1.3
ensemble	✓	12.69	8.9	1.7	12.4	29.1	1.7
Table 3: Ablation study on factors influencing the training of the object detector. Hierarchical level prediction is introduced as an auxiliary task for self-supervision. Mean Teacher self-training replaces the vanilla multi-round self-training and brings a smoother and more efficient training process. The weights of two learning targets, initial pseudo-labels and teacher predictions, are adaptively adjusted to build a more effective curriculum. All the three key designs are combined for the best overall performance of HASSOD.
Level Prediction	Mean Teacher	Adaptive Targets	AR	\( AR_S \)	\( Mask AR_M \)	\( AR_L \)	AP
20.2	9.2	30.4	42.5	5.7
✓	20.6	9.7	30.1	43.9	6.1
✓	✓	22.1	10.9	32.9	44.5	5.5
✓	✓	✓	22.4	11.3	32.9	45.0	6.3
Improvement over initial pseudo-labels. Although the initial pseudo-labels are produced by a frozen DINO backbone and they tend to be noisy and coarse, HASSOD is not upper-bounded by the quality of the fixed initial pseudo-labels or the pre-trained backbone. This is because of the following reasons: 1) While the initially discovered pseudo-labels are noisy, in the learning stage we train a detection model to learn common objects and their hierarchical relations for enhanced generalization to unseen images. By learning this detector, we boost the detection AR and AP from 8.9 and 1.7 (the last row in Table 2) to 20.6 and 6.1 (the second row in Table 3), respectively. Meanwhile, the pre-trained backbone features are adapted for the detection task in an end-to-end manner. 2) We further leverage Mean Teacher for continual self-enhancement, and gradually minimize the negative impact of noisy initial pseudo-labels. The evolving teacher detector and its features provide improved pseudo-labels to the student. Notably, we can directly read out the predictions from the teacher as the refined hierarchical pseudo-labels, instead of inefficiently running the clustering algorithm using the enhanced backbone. Consequently, we further improve the detection AR and AP to 22.4 and 6.3 (the last row in Table 3), respectively.
We present Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), an approach inspired by human visual perception that learns to detect objects and understand object composition in a self-supervised manner. HASSOD uses a hierarchical adaptive clustering strategy to propose a varying number of objects per image, and learns hierarchical levels of objects by analyzing geometric relations between objects. Mean Teacher self-training with adaptive targets facilitates the detector training process with smooth learning objectives and improved training efficiency. Empirical evaluation on recent large-scale image datasets Objects365, LVIS, and SA-1B demonstrates our significant improvement over prior self-supervised detectors. We detail the limitations and broader impacts of HASSOD in Appendix I.
This work was supported in part by the IBM-Illinois Discovery Accelerator Institute, NSF Grant #2106825, NIFA Award #2020-67021-32799, the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign through the NCSA Fellows program, the Illinois-Insp Per Partnership, and the Amazon Research Award. This work used NVIDIA GPUs at NCSA Delta through allocations CIS220014, CIS230012, and CIS230013 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program [3], which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296.
[1] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV, 2018. 15
[2] Irving Biederman. Recognition-by-components: A theory of human image understanding. Psychological review, 94(2):115, 1987. 1
[3] Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. ACCESS: Advancing innovation: NSF's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing, 2023. 11
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In CVPR, 2018. 5, 6, 15, 16, 21
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 1, 3, 4, 6, 15, 18, 20, 21
[6] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, 2014. 3
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 4, 6, 15, 20
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 4, 20
[9] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 32(9):1627-1645, 2009. 3
[10] Georgia Gkioxari, Ross Girshick, and Jitendra Malik. Actions and attributes from wholes and parts. In ICCV, 2015. 3
[11] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019. 3, 7, 8, 9, 14, 19
[12] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2009. 4
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6, 21
[14] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 16
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dólár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1
[16] Tarun Kalluri, Weiyao Wang, Heng Wang, Manmohan Chandraker, Lorenzo Torresani, and Du Tran. Open-world instance segmentation: Top-down learning with bottom-up supervision. arXiv preprint arXiv:2303.05503, 2023. 15
[17] Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open-world object proposals without learning to classify. IEEE Robotics and Automation Letters, 7(2):5453-5460, 2022. 15
[39] Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. TokenCut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. In CVPR, 2022. 3, 5, 7
[40] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detector2. https://github.com/facebookresearch/detectron2, 2019. 21
In this appendix, Section A first explains the deficiency of traditional MS-COCO AP evaluation in the self-supervised setting and reasons for adopting AR on more class-extensively-annotated datasets like LVIS. Sections B and C address potential concerns about unfair comparison with CutLER [38] regarding the training data and the detector architecture. Sections D, E, and F present additional ablation study regarding the threshold choice, computation cost, and patch size of the ViT backbone in our hierarchical adaptive clustering algorithm. Section G studies different behaviors of SAM and our proposed HASSOD in detecting parts of objects. Sections H and K present more comprehensive quantitative and qualitative evaluation results for completeness. Section I provides the failure cases of HASSOD and analyzes its current limitations. Section J describes the detailed hyper-parameter and implementation setup.
A Deficiency of MS-COCO AP Evaluation in Self-Supervised Object Detection
Traditionally, the Average Precision (AP) metric on the MS-COCO dataset [20] has been a gold standard for assessing the performance of supervised object detection and instance segmentation models, which are trained and evaluated on a fixed set of object categories pre-defined by human annotators. However, we find this metric misleading in the context of self-supervised object detection, where class labels are not available to the model and class-agnostic predictions are necessary. In this work, we advocate for evaluating Average Recall (AR) on datasets with extensive class annotations (e.g., LVIS [11]) as a more reliable metric for comparing different methods. In this section, we discuss the inherent deficiencies of MS-COCO AP evaluation using an illustrative example.
To demonstrate this problem objectively, we compare two previous methods, CutLER [38] and SAM [18]. Both models function as class-agnostic object detectors, but SAM, trained with human supervision, evidently outperforms CutLER in terms of detecting and segmenting objects, as depicted in Figure 6. Surprisingly, when evaluated on MS-COCO annotations, SAM attains a mere 6.7 AP, which is approximately half of CutLER's 12.3 AP. In this case, the AP metric is contradicting with the observed performance of the two models in terms of accurately detecting and segmenting as many objects as possible.
Figure 6: MS-COCO average precision (AP) evaluation may not accurately reflect the performance of class-agnostic object detectors. We compare two previous class-agnostic object detection/segmentation models, CutLER [38] and SAM [18]. Despite SAM, a supervised model, detecting more objects with superior localization than CutLER, it achieves only half the AP of CutLER.
The primary reason for this discrepancy lies in the annotations of MS-COCO. MS-COCO labels only 80 object categories, and model predictions are considered as true positives only if they fall within
these categories. Consequently, correct predictions for objects outside the 80 categories are unjustly deemed false positives and penalized by the AP metric, contradicting the objective of class-agnostic object detection and revealing the shortcomings of AP evaluation.
To correct these deficiencies in the evaluation metric, we need two changes: 1) Adopt a dataset with comprehensive annotations that include as many object categories as possible. If we substitute MS-COCO ground-truth annotations with LVIS, which labels over 1,200 object categories despite using the same images, the AP comparison is reversed: CutLER scores 4.5 AP on LVIS, while SAM attains a higher 6.1 AP. 2) Replace AP with the AR metric. Even with LVIS annotations, not all object categories are labeled in every image, resulting in certain valid object detection predictions still being penalized. Conversely, AR does not penalize such predictions and exhibits a more pronounced difference between CutLER and SAM (23.6 AR vs. 42.7 AR).
In summary, we advocate for AR on class-extensively-annotated datasets as the primary comparison metric, which can more accurately reflect the actual performance of class-agnostic object detectors with reduced bias. This choice of metric aligns with prior work on open-world detection [1, 17], segmentation [16, 36], and tracking [21] as well.
B Comparison of CutLER and HASSOD with Equal Training Data
As described in the main paper, HASSOD utilizes a ResNet-50 backbone initialized with DINO [5] weights pre-trained in a self-supervised manner on ImageNet [7], while subsequent training is conducted on MS-COCO [20] images. We choose MS-COCO for its compact size and richness in objects per image. Due to limited computation resources, we are unable to perform HASSOD training on ImageNet, and we leave large-scale training as one interesting future direction. This distinction in the training dataset may lead to concerns regarding the fairness of comparison with prior work, such as CutLER [38], which trains exclusively on ImageNet data. However, it is important to note that using both ImageNet and MS-COCO data in HASSOD does not grant HASSOD additional benefit compared with CutLER for two main reasons: 1) The two datasets are leveraged in separate stages and not in a blended manner. Specifically, ImageNet data are only used in the DINO pre-training stage, while MS-COCO is employed for detector training. 2) ImageNet contains approximately  $5 \times$  as many images as MS-COCO. Thus, if ImageNet is employed in the detector training stage, as is the case with CutLER, this would actually lead to a stronger detector.
Table 4: Comparison between CutLER [38] and HASSOD concerning the training image dataset. Although both CutLER and HASSOD leverage DINO [5] weights pre-trained on ImageNet, employing MS-COCO images in the detector training stage does not lead to superior performance. CutLER trained on ImageNet surpasses CutLER trained on MS-COCO across all metrics. Simultaneously, HASSOD outperforms both CutLER models, despite using MS-COCO training data and requiring fewer training iterations.
Method	Training Images	Training Iterations	AR	\( AR_S \)	Mask \( AR_M \)	\( AR_L \)	AP
CutLER [38]	MS-COCO [20]	160,000	17.3	6.2	24.2	46.1	5.8
CutLER [38]	ImageNet [7]	160,000	18.8	7.2	27.6	46.6	6.2
HASSOD (Ours)	MS-COCO [20]	40,000	22.4	11.3	32.9	45.0	6.3