SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to address these concerns more effectively and ensure an equal usage of training data, we conduct an additional experiment. Specifically, we train a Cascade Mask R-CNN [4] detector using the CutLER approach on MS-COCO images for one round (160,000 iterations), with the ResNet-50 backbone initialized with DINO pre-trained weights. We adopt CutLER's original implementation, with the sole modification being the use of MS-COCO images for training. We compare this model with a CutLER model trained on ImageNet for one round and our HASSOD model trained on MS-COCO. The evaluation is conducted against MS-COCO+LVIS annotations, as described in Section 4.5. The results are presented in Table 4.
By comparing the two CutLER models trained with MS-COCO and ImageNet, we observe that utilizing distinct datasets for backbone pre-training and detector training does not universally improve performance. The CutLER model, exclusively trained on ImageNet, surpasses its counterpart that uses
ImageNet for DINO pre-training and MS-COCO for detector training, exhibiting gains of 1.5 Mask AR and 0.4 Mask AP. Meanwhile, HASSOD, despite being trained on the smaller MS-COCO dataset and for a shorter duration, outperforms CutLER by 3.6 Mask AR. This remarkable performance is attributed to HASSOD's comprehensive object coverage in images and its efficient usage of training data.
C Additional Results on Mask R-CNN
In HASSOD, we train a Cascade Mask R-CNN [4] object detection and instance segmentation model. We choose this architecture following CutLER [38] and we ensure a fair comparison with this prior work. In fact, HASSOD can be applied on different detector architectures. As an example, we also train a Mask R-CNN [14] and compare its LVIS performance with CutLER in Table 5. With the same detector architecture, HASSOD produces a better detector than CutLER. More impressively, our Mask R-CNN (weaker architecture) outperforms CutLER's Cascade Mask R-CNN (stronger architecture) on LVIS, highlighting the advantage of our approach.
Table 5: Comparison between CutLER [38] and HASSOD with different detector architectures on the LVIS dataset. When training models of the same architecture, HASSOD outperforms CutLER (row 1 vs. 2, row 3 vs. 4). Notably, HASSOD has a stronger AR even with a weaker architecture (row 2 vs. 3) as compared with CutLER.
Architecture	Method	Box	Mask
AR	\( AR_S \)	\( AR_M \)	\( AR_L \)	AP	AR	\( AR_S \)	\( AR_M \)	\( AR_L \)	AP
Mask R-CNN	CutLER [38]	20.7	10.4	33.3	52.0	4.1	18.5	9.6	29.1	44.9	3.4
HASSOD (Ours)	23.8	13.5	38.3	50.9	4.3	21.5	11.8	35.1	46.6	4.1
Cas. Mask R-CNN	CutLER [38]	23.6	13.1	36.2	55.6	4.5	20.2	11.3	31.1	46.2	3.6
HASSOD (Ours)	26.9	15.6	42.2	56.9	4.9	22.5	12.7	36.1	47.8	4.2
D Choosing Thresholds for Hierarchical Adaptive Clustering
When determining the merging thresholds  $\{\theta_i^{\mathrm{merge}}\}$ , we mainly consider our computational constraints and empirical observations. The decision of merging thresholds is not made based on validation performance (Table 2), ensuring that HASSOD is fully self-supervised.
Figure 7: Relation between the number of pseudo-labels per image after post-processing (y-axis) and the merging threshold  $\theta_{i}^{\mathrm{merge}}$  (x-axis). When  $\theta_{i}^{\mathrm{merge}} \geq 0.5$ , the number of pseudo-labeled masks grows rapidly. Empirically, we find that when the number of masks per image exceeds 20, generating, loading, and transforming such pseudo-labels becomes a major bottleneck in HASSOD. Therefore, we choose three thresholds  $\{\theta_{i}^{\mathrm{merge}}\} = \{0.1, 0.2, 0.4\}$ , mainly guided by a computational consideration.
Guidance by number of pseudo-masks. Our choice for  $\{\theta_i^{\mathrm{merge}}\}$  is primarily guided by the number of pseudo-label masks produced per image. Figure 7 shows the relationship between the number of masks per image and different thresholds. When  $\theta^{\mathrm{merge}} \geq 0.5$ , the number of masks per image escalates rapidly. This steep increase incurs significant computational costs, both during the initial generation of pseudo-labels and the subsequent data loading and pre-processing procedures during model training. To strike a balance between computational efficiency and the desired mask granularity, thresholds of  $\{\theta_i^{\mathrm{merge}}\} = \{0.1, 0.2, 0.4\}$  are chosen.
Table 6: Generalizability of the merging thresholds  $\{\theta_i^{\mathrm{merge}}\}$ . With the same merging threshold, the number of generated pseudo-labels is not significantly changing across different image datasets.
θmerge	Number of pseudo-labels per image
MS-COCO [20]	Objects365 [27]	SA-1B [18]
0.1	2.58	3.49	2.91
0.2	4.20	5.78	4.88
0.4	11.61	12.15	12.70
Threshold generalization across datasets. Another noteworthy observation is the generalizability of these thresholds across various datasets. In Table 6, we present the number of generated pseudo-labels per image on three datasets. With the merging threshold  $\theta_{i}^{\mathrm{merge}}$  fixed, the number of generated labels is relatively stable, regardless of the source image dataset. Therefore, our pre-set thresholds are generalizable and require no further tuning when transferred to other datasets. Meanwhile, our detection model was trained on MS-COCO images with pseudo-labels generated using our pre-set  $\{\theta_{i}^{\mathrm{merge}}\}$ , and could generalize well to other datasets in a zero-shot manner, as shown in Table 1. This fact shows that the thresholds  $\{\theta_{i}^{\mathrm{merge}}\}$  are effective regardless of evaluation datasets.
E Computational Costs in Hierarchical Adaptive Clustering
We adopt the hierarchical adaptive clustering strategy to generate our initial pseudo-labels. In this section, we provide additional details regarding computation costs in this procedure.
As shown in Figure 3, the hierarchical adaptive clustering contains four steps "merge," "post-process," "ensemble," and "split." Among them, the "merge" step accounts for the major computation costs. We can analyze its time complexity: Suppose we have  $n$  patches in the beginning, then there are at most  $n$  merging steps before stopping. Each merging step requires retrieving the most similar pair of adjacent regions. The collection of adjacent pairs has size at most  $O(n)$ , and each operation requires time  $O(\log n)$  if this collection is organized as a balanced binary tree. Therefore, the merging process has time complexity  $O(n \log n)$ . In our practice, the input image has resolution  $480 \times 480$ , so  $n = \frac{480}{8} \times \frac{480}{8} = 3,600$ . This time complexity is affordable.
More concretely, we list the time costs in the hierarchical adaptive clustering on our computation platform in Table 7. Since the procedure is learning-free, we can parallelize the processing of images using more than one worker. On our computation nodes equipped with 4 NVIDIA A100 GPUs, we can reduce the processing time to 1.59 sec/image. With 4 such nodes, we can complete the pseudo-label generation for MS-COCO train and unlabeled splits (0.24 million images) in  $\frac{1.59 \times 0.24 \times 10^6}{4 \times 86400} \approx 1$  day.
Table 7: Time costs in the steps of the hierarchical adaptive clustering. Notably, with multiple parallel workers, we can reduce the total processing time of MS-COCO [20] images to one day.
Step	Time Cost (sec/image)	Workers	Parallelized Cost (sec/image)
Merge and Post-Process Ensemble and Split	11.7	8	1.46
2.1	16	0.13
Total	13.8	-	1.59
F Impact of Patch Size in Hierarchical Adaptive Clustering
We use the DINO [5] pre-trained ViT-B/8 backbone to generate the initial pseudo-labels through our hierarchical adaptive clustering. We observe that the small patch size  $8 \times 8$  leads to better pseudo-label quality. In Table 8, we compare the mask quality of pseudo-labels generated by ViT-B/8 (patch size  $8 \times 8$ ) vs. ViT-B/16 (patch size  $16 \times 16$ ). For a fair comparison, we use  $480 \times 480$  input resolution for ViT-B/8 and  $960 \times 960$  for ViT-B/16, so that they have the same number of initial patches. With the same merging threshold  $\theta^{\text{merge}}$ , ViT-B/16 leads to slightly fewer labels per image, and the quality is significantly worse than ViT-B/8, especially for small and medium objects. Therefore, we apply ViT-B/8 in our experiments for its localized visual features and subsequent high-quality pseudo-labels.
Table 8: Comparison between DINO ViT backbones with different patch sizes  $8 \times 8$  and  $16 \times 16$ . The backbone with the smaller patch size leads to higher-quality initial pseudo-labels in the hierarchical adaptive clustering procedure.
DINO Backbone	θmerge	Labels per Img	AR	ARS	Mask ARM	ARL	AP
ViT-B/8	0.1	2.58	4.1	0.6	5.1	15.6	1.8
0.2	4.20	5.3	0.9	7.0	18.4	1.8
0.4	11.61	7.8	1.7	12.1	22.1	1.3
ViT-B/16	0.1	1.97	3.0	0.3	2.3	14.6	1.1
0.2	3.19	3.8	0.4	3.3	17.4	1.2
0.4	10.15	5.7	0.9	6.6	21.7	1.3
G Different Behaviors of SAM and HASSOD in Object Part Detection
Segment Anything Model (SAM) [18], a supervised segmentation model, has demonstrated its ability in creating high-quality, fine-grained segmentation of images. Directly comparing the performance between SAM and HASSOD would be unbalanced, considering SAM requires extensive human-labeled images for training, while HASSOD operates entirely under self-supervision. Despite this, it remains intriguing to understand their different behaviors. In this section, we delve into a qualitative comparison between SAM and our proposed HASSOD approach, paying particular attention to their respective abilities to detect constituent parts of whole objects. Figure 8 presents a visualization for this comparison.
Both SAM and HASSOD can perform fine-grained segmentation within complex scenes, successfully detecting individual object parts that constitute a whole entity. However, a clear distinction arises in their respective approaches towards the segmentation of these object parts. For scenes in which objects follow a grid pattern, where the whole entity is partitioned by regular boundary lines into semantically similar pieces, SAM tends to perceive each grid as a distinct object. Conversely, HASSOD adheres to a holistic perspective for such cases, grouping the grid parts together even at the subpart level, due to their semantically analogous features.
In particular, HASSOD excels at distinguishing object parts that contain different contents within whole objects. This skill of HASSOD is especially advantageous in certain real-world applications. For example, in medical imaging, HASSOD's fine-grained segmentation can potentially assist in identifying and separating different tissues or anatomical structures within a scan. Additionally, in the manufacturing industry, HASSOD could be used in quality control to check and compare individual components of an assembled product. While both segmentation strategies of SAM and HASSOD are valid, HASSOD's unique ability to distinguish semantically different parts within objects provides distinct advantages in a wide range of practical scenarios. Comprehensively quantifying and analyzing such an ability is crucial for advancement of self-supervised object detection and segmentation approaches, and we consider it as an important future direction.
Figure 8: Analysis of different behaviors in object part detection between supervised SAM [18] and self-supervised HASSOD through qualitative visualization. While both models generate fine-grained segmentation, they exhibit distinct preferences concerning object parts. SAM inclines towards separating objects following a grid pattern (e.g., comforters, keyboards, tiles), whereas HASSOD discriminates objects into semantically diverse parts (e.g., eyes and beard of a cat, handle and button of a cabinet).
H Additional Evaluation Results for Completeness
As an addition to Table 1 in the main paper, we present a comprehensive evaluation of HASSOD on the Objects365 [27], LVIS [11], and SA-1B [18] datasets in Table 9. For a robust measure of performance, we incorporate the standard deviation, estimated from three independently trained models.
Table 9: Comprehensive evaluation results of HASSOD across three datasets. We set the maximum number of predictions per image to 1,000. The number superscript on AR (e.g.,  $\mathrm{AR}^{10}$ ) denotes the number of most confident predictions taken into account when computing AR. The subscript on AP (e.g.,  $\mathrm{AP}_{50}$ ) represents the Intersection-over-Union (IoU) threshold utilized when matching predictions with ground truth labels. Size-specific metrics for small, medium, and large objects are indicated by the subscripts  $S$ ,  $M$ , and  $L$ , respectively. We include the standard deviation, estimated from three independent runs.
Dataset	\( \mathrm{AR}^{10} \)	\( \mathrm{AR}^{100} \)	\( \mathrm{AR}^{1000} \)	\( \mathrm{AR}_S \)	\( \mathrm{AR}_M \)	\( \mathrm{AR}_L \)	AP	\( \mathrm{AP}_{50} \)	\( \mathrm{AP}_{75} \)	\( \mathrm{AP}_S \)	\( \mathrm{AP}_M \)	\( \mathrm{AP}_L \)
Objects365 [27]	Box (Object Detection)
15.20±0.10	36.63±0.06	39.03±0.06	21.40±0.10	40.43±0.12	52.10±0.17	10.97±0.15	20.33±0.31	10.27±0.15	2.90±0.10	10.37±0.25	19.47±0.25
LVIS [11]	10.58±0.07	25.01±0.02	26.87±0.03	15.64±0.05	42.22±0.08	56.90±0.17	4.94±0.09	9.03±0.09	4.75±0.10	2.82±0.01	7.90±0.14	12.19±0.20
SA-1B [18]	5.52±0.02	23.92±0.03	29.02±0.18	13.34±0.27	25.12±0.21	43.79±0.12	15.47±0.08	26.20±0.09	15.90±0.03	5.39±0.06	15.06±0.13	21.81±0.08
Mask (Instance Segmentation)
LVIS [11]	9.69±0.06	21.11±0.05	22.50±0.01	12.68±0.01	36.14±0.11	47.79±0.17	4.21±0.20	7.99±0.25	3.95±0.24	1.88±0.19	6.96±0.21	13.67±0.18
SA-1B [18]	5.27±0.01	21.62±0.07	25.99±0.22	12.90±0.37	22.76±0.24	38.33±0.21	13.85±0.09	24.78±0.08	13.67±0.11	4.09±0.04	13.45±0.10	20.36±0.07
I Limitations and Broader Impacts
Limitations. Due to the self-supervised nature of HASSOD, the learned object hierarchical levels may not be perfectly aligned with human perception. This mismatch may lead to over-segment or under-segment of objects in real-world applications.
Figure 9: Failure cases of HASSOD. Top: Overlapping and similar objects are hard to distinguish. Middle: The whole object comprising diverse parts is not identified. Bottom: Text contents are not precisely localized.
We observe that HASSOD performs unsatisfactorily in certain scenes due to this lack of human supervision. Figure 9 provides a visualization of the failure cases. In the first row, HASSOD mistakenly treats the lower parts of the two players as a single coherent object. Due to their similar color and texture, multiple overlapping instances of the same class can sometimes be perceived as one object. In the second row, HASSOD fails to predict a mask that encompasses the entire motorcycle. This object consists of highly heterogeneous parts, making it challenging to recognize them as components of a single entity. In the third row, HASSOD fails to detect the text "Koelnmesse," and the boundaries for other text are not clear. We also observe that such errors are not unique to HASSOD; they appear in prior self-supervised object detection methods like CutLER as well. We believe that further human supervision would be necessary for correcting these mistakes.
Broader impacts. Detecting object parts with self-supervision may be beneficial to real-world applications including robotic manipulation and inspection. As a general object detection method, we share risks associated with applying recognition models such as abuse of surveillance systems.
J Hyper-Parameters and Implementation Details
In the initial pseudo-label generation process, we use a frozen ViT-B/8 model [8] pre-trained on unlabeled ImageNet [7] by DINO [5], a self-supervised representation learning method, to extract visual features of train and unlabeled images in MS-COCO [20]. Following prior work CutLER [38], we resize the resolution of each image to  $480 \times 480$ , leading to  $60 \times 60$  patches as initial regions. The merging process stops at three thresholds  $\theta_1^{\mathrm{merge}} = 0.4$ ,  $\theta_2^{\mathrm{merge}} = 0.2$ ,  $\theta_3^{\mathrm{merge}} = 0.1$ , and results from these three thresholds are ensembled after post-processing. The post-processing steps include Conditional Random Field (CRF) [19] and filling the holes in each mask. We also filter out low-quality masks that 1) have an Intersection-over-Union (IoU) smaller than 0.5 before and after CRF, 2) are smaller than 100 pixels, or 3) contain more than two corners of the image (which are likely
background). These post-processing steps are also used in prior work including CutLER [38]. After assembling results from the three thresholds  $\{\theta_i^{\mathrm{merge}}\}_{i = 1}^3$ , we identify the hierarchical levels of each object mask based on the coverage relation analysis. The coverage threshold is set to  $\theta^{\mathrm{cover}}\% = 90\%$ .
In the detector training stage, we train a Cascade Mask R-CNN [4] with a ResNet-50 [13] backbone on the same MS-COCO [20] images. The backbone is initialized from DINO [5] self-supervised pre-training. Our hyper-parameter setting for Mean Teacher mostly follows the practice of Unbiased Teacher [22]. The whole training process starts with a "burn-in" stage, during which the student model is only trained on the initial pseudo-labels with a fixed learning rate 0.01 and fixed loss weights. After the burn-in stage, the teacher model is introduced, and we gradually adjust the learning rate from 0.01 to 0, the loss weight in the label-to-student branch from 1.0 to 0.0, and the loss weight in the teacher-to-student branch from 2.0 to 3.0, all following a cosine schedule. The whole training process spans 40,000 iterations with a batch size of 16 images. The training is performed on  $4 \times$  NVIDIA A100 GPUs. Our code is developed based on PyTorch [24] and Detectron2 [40].
K Additional Qualitative Results
In this section, we present visualization of detection results by CutLER [38] and HASSOD on Objects365 [27] in Figure 10 and SA-1B [18] in Figure 11 (see next pages). Qualitative results on LVIS has been included in the main paper, Figure 5.
Figure 10: Qualitative results on Objects365 [27]. In each row, we show detection results of prior state-of-the-art self-supervised object detector CutLER, whole objects predicted by HASSOD, and all object predicted by HASSOD.
Figure 11: Qualitative results on SA-1B [18]. In each row, we show detection results of prior state-of-the-art self-supervised object detector CutLER, whole objects predicted by HASSOD, and all object predicted by HASSOD.