SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shengcao Cao $^{1}$  Dhiraj Joshi $^{2}$  Liang-Yan Gui $^{1}$  Yu-Xiong Wang $^{1}$
<sup>1</sup>University of Illinois at Urbana-Champaign <sup>2</sup>IBM Research
$^{1}$  {cao44,lgui,yxw}@illinois.edu  $^{2}$  djoshi@us.ibm.com
The human visual perception system demonstrates exceptional capabilities in learning without explicit supervision and understanding the part-to-whole composition of objects. Drawing inspiration from these two abilities, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), a novel approach that learns to detect objects and understand their compositions without human supervision. HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image. Furthermore, HASSOD identifies the hierarchical levels of objects in terms of composition, by analyzing coverage relations between masks and constructing tree structures. This additional self-supervised learning task leads to improved detection performance and enhanced interpretability. Lastly, we abandon the inefficient multi-round self-training process utilized in prior methods and instead adapt the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. Through extensive experiments on prevalent image datasets, we demonstrate the superiority of HASSOD over existing methods, thereby advancing the state of the art in self-supervised object detection. Notably, we improve Mask AR from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B. Project page: https://HASSOD-NeurIPS23.github.io.
The development of human visual perception is remarkable for two key abilities: 1) Humans begin learning to perceive objects in their environment through observation alone [25], without needing to learn the names of these objects from external supervision. 2) Moreover, human perception operates in a hierarchical manner, enabling individuals to recognize the part-to-whole composition of objects [2, 23]. These characteristic capabilities offer valuable insights into the learning processes of object detectors, which still heavily rely on the availability and quality of fine-grained training data. For example, the state-of-the-art detection/segmentation model, Segment Anything Model (SAM) [18], is developed on a dataset of 11 million images and 1 billion object masks. It remains an open question how to effectively learn to detect objects and recognize their compositions from even larger-scale datasets (e.g., LAION-5B [26]) without such object-level annotations.
In prior work on self-supervised object detection [37, 38], a two-stage discover-and-learn paradigm is adopted: 1) Self-supervised visual representations [5, 15] are obtained, and a saliency-based method is employed to extract the most prominent one or few objects. 2) Subsequently, an object detector is trained based on these pseudo-labels, sometimes involving multiple rounds of self-training for refinement. However, despite such attempts to eliminate the need for external supervision, several weaknesses persist in these approaches: 1) Narrow coverage of objects. The focus on only one or few objects per image in previous methods undermines their ability to fully exploit the learning signals in
Figure 1: Fully self-supervised object detection and instance segmentation on prevalent image datasets. Our approach, HASSOD, demonstrates a significant improvement over the previous state-of-the-art method, CutLER [38], by discovering a more comprehensive range of objects. Moreover, HASSOD understands the part-to-whole object composition like humans do, while previous methods cannot.
natural scene images containing dozens of objects, such as those in the MS-COCO dataset [20]. This narrow focus also restricts the capability of these methods to accurately detect and segment multiple objects within an image. 2) Lack of composition. Prior work often overlooks the composition of objects, neglecting the identification of hierarchical levels for whole objects, part objects, and subpart objects (e.g., considering an image of a bicycle; the bicycle is a whole object, its wheels and handles are parts, and the spokes and tires are subparts). This oversight not only limits the interpretability of learned object detectors, but also hinders the model's ability to tackle the intrinsic ambiguity in the task of segmentation. 3) Inefficiency. The reliance on multi-round self-training in earlier methods can result in inefficient and non-smooth training processes, which further constrains the potential of self-supervised object detection and comprehension of object composition.
Inspired by the unsupervised, hierarchical human visual perception system, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), aiming to address these limitations mentioned above and better harness the potential of self-supervised object detection, as depicted in Figure 1. First, unlike previous methods that limit their focus to one or few prominent objects per image, HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks, based on self-supervised visual representations. By adjusting the threshold for terminating the clustering process, HASSOD is capable of effectively determining the appropriate number of objects per image, thus better leveraging the learning signals in images with multiple objects.
The second key component of HASSOD is its ability to identify hierarchical levels of objects in terms of composition. By analyzing the coverage relations between masks and building tree structures, our approach successfully classifies objects as whole objects, part objects, or subpart objects. This novel self-supervised learning task not only improves detection performance, but also enhances the interpretability and controllability of the learned object detector, a property that prior self-supervised detectors lack. Therefore, HASSOD users can comprehend how the detected whole objects are assembled from smaller constituent parts. Simultaneously, they can control HASSOD to perform detection at their preferred hierarchical level, thereby catering more effectively to their needs.
Finally, HASSOD abandons the multi-round self-training used in previous methods which lacks efficiency and smoothness. Instead, we take inspiration from the Mean Teacher [22, 31] framework in semi-supervised learning, employing a teacher model and a student model that mutually learn from each other. This innovative adaptation facilitates a smoother and more efficient training process, resulting in a more effective self-supervised object detection approach.
In summary, the key contributions of HASSOD include:
- A hierarchical adaptive clustering strategy that groups regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image and effectively discover more objects from natural scenes.
- The ability to identify hierarchical levels of objects in terms of composition (whole/part/subpart) by analyzing coverage relations between masks and building tree structures, leading to improved detection performance and enhanced interpretability.
- A novel adaptation of the Mean Teacher framework from semi-supervised learning, which replaces the multi-round self-training in prior methods, leading to smoother and more efficient training.
- State-of-the-art performance in self-supervised object detection, enhancing Mask AR from 20.2 to 22.5 on LVIS [11], and from 17.0 to 26.0 on SA-1B [18]. Remarkably, these results are achieved through training with only  $\mathbf{1} / \mathbf{5}$  of the images and  $\mathbf{1} / \mathbf{12}$  of the iterations required by prior work.
Unsupervised object detection/discovery. Identifying and locating objects in images without using any human annotations is a challenging task, as it requires learning the concept of objects from image data without any external supervision. OSD [33] formulates this task as an optimization problem on a graph, where the nodes are object proposals generated by selective search, and the edges are constructed based on visual similarities. rOSD [34] improves the scalability of OSD with a saliency-based region proposal algorithm and a two-stage strategy. LOD [35] formulates unsupervised object discovery as a ranking optimization problem for improved computation efficiency. Following the observation that DINO [5], a self-supervised pre-training method, can segment the most prominent object in each image, LOST [29], FOUND [30], and FreeSOLO [37] train object detectors using saliency-based pseudo-labels. TokenCut [39] and CutLER [38] also use self-supervised representations, but generate pseudo-labels by extending Normalized Cuts [28]. Saliency-based region proposal and Normalized Cuts are both focused on the prominent objects in each image, and usually only propose one or few objects per image. Different from these approaches, HASSOD produces initial pseudo-labels using a hierarchical adaptive clustering strategy, which can adaptively determine the number of objects depending on the image contents.
Object detection by parts. Detecting objects by identifying their composing parts has been widely studied in computer vision. Deformable Parts Model (DPM) [9] is a seminal approach that utilizes discriminatively trained part-based models for object detection, which effectively models complex object structures and improves over monolithic detectors. A following method [6] not only detects the objects, but also simultaneously represents them using body parts, highlighting the importance of both holistic models and part-based representations. This idea is extended by leveraging both whole object and part detections to infer human actions and attributes [10], suggesting the advantage of a combined approach. In this work, we revisit this classic idea of representing and detecting whole objects as well as their parts in the context of self-supervised learning.
In this section, we introduce the learning process in our proposed approach, Hierarchical Adaptive Self-Supervised Object Detection (HASSOD). Following prior work on unsupervised object detection [29, 30, 37-39], HASSOD adopts a two-stage discover-and-learn process to learn a self-supervised object detector, as illustrated in Figure 2. In the first stage, we discover objects from unlabeled images using self-supervised representations, and generate a set of initial pseudo-labels. Then in the second stage, we learn an object detector based on the initial pseudo-labels, and smoothly refine the model by self-training. The first stage is based on pre-trained, fixed visual features, and the second stage learns an object detector to improve over the fixed visual features and pseudo-labels. In the following subsections, we describe the three core components of HASSOD in detail.
3.1 Hierarchical Adaptive Clustering
In the first stage, HASSOD creates a set of pseudo-labels as the initial self-supervision source. We propose a hierarchical adaptive clustering strategy to discover object masks as pseudo-labels, using only unlabeled images and a frozen self-supervised visual backbone. Figure 3 provides an overview of this procedure. Our hierarchical adaptive clustering algorithm extends agglomerative
Figure 2: Two-stage discover-and-learn process in HASSOD. Stage 1 uses a frozen, self-supervised DINO [5] ViT backbone to discover initial pseudo-labels from unlabeled images. Stage 2 learns an object detector to improve over the pre-trained features and initial pseudo-labels.
Figure 3: Hierarchical adaptive clustering and hierarchical levels of objects. The procedure of creating initial pseudo-labels for training the object detector without any human annotations includes the following steps: (Initialize) Visual features are extracted from the given image by a ViT pre-trained with DINO [5], and each  $8 \times 8$  patch is initialized as one individual region. (Merge) Adjacent regions with the highest feature similarities are progressively merged into object masks, until the pre-set thresholds  $\theta_{i}^{\text{merge}}$  are reached. (Post-Process) Object masks are selected and refined using simple post-processing techniques. (Ensemble) Results from multiple thresholds  $\{\theta_{i}^{\text{merge}}\}_{i=1}^{3}$  are combined to ensure better coverage of potential objects. (Split) Analysis of coverage relations divides objects into three hierarchical levels: whole, part, and subpart. The example on the right illustrates the tree structure of object composition: The whole aircraft is composed of an upper and a lower part. The upper part further consists of a left wing, a right wing, and a person standing on it.
clustering [12], grouping adjacent image patches into semantically coherent masks based on the similarity of self-supervised visual representations. More specifically, we use a frozen ViT-B/8 model [8] pre-trained on unlabeled ImageNet [7] by DINO [5], a self-supervised representation learning method, to extract visual features. For each image, we take the feature map generated by this model at its final Transformer layer [32]. Each spatial element in the feature map corresponds to a  $8 \times 8$  patch in the original image.
To initiate the hierarchical adaptive clustering process, we treat each patch as an individual region. We then compute the pairwise cosine similarity between the features of adjacent regions to measure their closeness in the semantic feature space. The regions are gradually merged into masks that represent objects by iteratively performing the following steps: 1) Identify the pair of adjacent regions with the highest feature similarity. 2) If the similarity is smaller than the pre-set threshold  $\theta^{\mathrm{merge}}$ , stop the merging process. 3) Merge the two regions, and compute the feature of the merged region by averaging all the patch-level features within it. 4) Update the pairwise similarity between this newly merged region and its neighbors. This merging process is visualized in Figure 3, columns 1-3.
Once the merging process is complete, we perform a series of automated post-processing steps to refine and select the masks, including Conditional Random Field (CRF) [19] and filtering out masks that are smaller than 100 pixels or contain more than two corners of the image. These steps are based on standard practices in previous work [38] and require no manual intervention. Our hierarchical adaptive clustering strategy effectively groups regions into object masks based on self-supervised visual representations, adaptively determining the appropriate number of objects per image. In
images containing multiple objects with heterogeneous semantic features, the merging process stops earlier, resulting in a larger number of regions corresponding to different objects. Conversely, in highly homogeneous images, more regions are merged, leading to fewer object masks. This adaptive approach enables HASSOD to cover more objects for self-supervised learning, rather than being limited by one or a few prominent objects in each image in prior work [38, 39].
In practice, we are not restricted to one single fixed threshold  $\theta^{\mathrm{merge}}$  to determine the stopping criterion for the clustering process. Instead, we find it beneficial to ensemble results from multiple (e.g., 3) pre-set thresholds  $\{\theta_i^{\mathrm{merge}}\}_{i=1}^3$ . When the currently highest feature similarity reaches one of these thresholds, we record the derived object masks from the merged regions at that step. Utilizing multiple thresholds allows us to capture objects of various sizes and at different hierarchical levels of composition, enabling a more comprehensive coverage of objects in scene images. The post-processing and ensemble are visualized in Figure 3, columns 4-5.
3.2 Hierarchical Level Prediction
In the following second stage, HASSOD learns an object detection and instance segmentation model, e.g., Cascade Mask R-CNN [4], using the initial pseudo-labels generated in the first stage. By training on such pseudo-labels, the model learns to recognize common objects across different training images, and thus achieves enhanced generalization to images which the model has not seen during training.
In addition to the standard object detection objective, we aim to equip our detector with the ability to understand the hierarchical structure among objects and their constituent parts. In HASSOD, we incorporate the concept of hierarchical levels into object masks by leveraging the coverage relations between them. Formally, we say mask A is covered by mask B when three conditions are satisfied (with respect to a pre-set coverage threshold  $\theta^{\mathrm{cover}}\%$  : 1) More than  $\theta^{\mathrm{cover}}\%$  of pixels in mask A are also in mask B. 2) Less than  $\theta^{\mathrm{cover}}\%$  of pixels in mask B are in mask A. 3) Mask B is the smallest among all masks satisfying the previous two conditions. Intuitively, if mask B covers mask A, it suggests that A is a part of B and B is at a higher level than A. If we consider A and B as tree nodes, A should be a child of B. Using all such coverage relations, we can construct a forest of trees that contain all masks in an image. Ultimately, the roots of all trees in this image are considered as "whole" objects, their direct children are "part" objects, and all the remaining descendants are "subpart" objects. An example is shown on the right side of Figure 3.
After identifying the hierarchical levels of object masks in the pseudo-labels, we attach a new classification head to the object detector for level prediction, which classifies each predicted object as a whole object, a part object, or a subpart object. This new component enables HASSOD to model object composition effectively, resulting in improved object detection performance and enhanced interpretability compared with previous self-supervised object detection methods. The hierarchical level prediction head is added alongside the existing foreground/background classification head, box regression head, and mask prediction head. Subsequently, we train the object detector using the initial set of object mask pseudo-labels obtained from the hierarchical adaptive clustering process, as well as the additional level prediction task.
3.3 Mean Teacher Training with Adaptive Targets
Notably, the initial pseudo-labels derived in the first stage contain noise and are not perfectly aligned with real objects. To improve over such noisy pseudo-labels, prior work [37, 38] usually employs multi-round self-training to refine the model, i.e., using a well-trained detector to re-generate pseudolabels and re-train a new detector. For the first time, HASSOD refines the object detector efficiently and smoothly by adapting the Mean Teacher learning paradigm [22, 31] from semi-supervised learning to the fully self-supervised setting.
Before introducing our innovative adaptation of Mean Teacher in the self-supervised setting, we first briefly summarize the mutual-learning process in Mean Teacher (see Figure 4). Mean Teacher employs two models, a teacher and a student, which learn from each other. The teacher takes weakly-augmented, unlabeled images as input and provides detection outputs as training targets for the student. The student's weights are updated to minimize the discrepancy between its predictions and the targets given by the teacher on the same unlabeled images but with strong augmentation. In the semi-supervised setting, the student receives supervision from two sources simultaneously. One source is the "teacher-to-student" branch mentioned above, and the other is the "label-to-student" branch
Figure 4: Mean Teacher self-training with adaptive targets in HASSOD. Two detectors of the same architecture, the teacher and the student, learn from each other to improve over the initial pseudo-labels. The teacher is updated as the exponential moving average (EMA) of the student. The student receives supervision from two branches: The teacher-to-student branch (top) encourages the student to mimic the teacher's predictions; the label-to-student branch (bottom) minimizes the discrepancy between the student's predictions and the initial pseudo-labels. During training, our proposed adaptive target strategy increases the weight for the teacher-to-student branch,  $\alpha_{\mathrm{teacher}}$ , and decreases the weight for the label-to-student branch,  $\alpha_{\mathrm{label}}$ , since the teacher becomes a more and more reliable self-supervision source compared with the initial pseudo-labels.
where the student learns from images with ground-truth labels. Both branches compute standard detection losses (e.g., bounding box classification and regression), and the student is optimized to minimize the total loss. The teacher's weights are an exponential moving average of the student's weights, ensuring smooth and stable training targets.
In HASSOD, we do not have any labeled images from human supervision but instead utilize two sources of self-supervision. One source is the initial pseudo-labels obtained from our hierarchical adaptive clustering, which functions similarly to the labels-to-student branch in the semi-supervised setting. The other source is the detection predictions made by the teacher model, which corresponds to the teacher-to-student branch in Mean Teacher. Different from standard Mean Teacher, our method employs adaptive training targets, as we gradually adjust the loss weights for the two branches. This is because the initial pseudo-labels may not effectively cover all possible objects, while the teacher model will progressively improve as a better source of supervision. Consequently, during Mean Teacher self-training, we continuously decrease the loss weight  $\alpha_{\mathrm{label}}$  for the branch that uses the initial pseudo-labels and increase the loss weight  $\alpha_{\mathrm{teacher}}$  for the branch based on the teacher's predictions, following a cosine schedule.
In this section, we conduct extensive experiments to evaluate HASSOD in comparison with previous methods. We first describe the training details and efficiency in Section 4.1. Section 4.2 introduces the datasets and metrics used for evaluation. Section 4.3 presents our main results of self-supervised object detection and instance segmentation. Section 4.4 provides some qualitative results and analysis. Section 4.5 conducts further experiments to verify the effects of each component in HASSOD. Additional quantitative and qualitative results are included in the appendix.
4.1 Data-Efficient and Computation-Efficient Training
We train a Cascade Mask R-CNN [4] with a ResNet-50 [13] backbone on MS-COCO [20] images. The backbone is initialized from DINO [5] self-supervised pre-training. We use both the train and unlabeled splits of MS-COCO, totaling to about 0.24 million images. Notably, this amount of images is only 1/5 of ImageNet used by prior work CutLER [38]. Compared with ImageNet-like [7] iconic images, images in MS-COCO are mostly captured in complex scenes containing multiple objects with diverse layouts and compositions. Therefore, each image offers richer learning resources for object detectors, enabling effective detector training with significantly fewer images. The whole
training process spans 40,000 iterations, taking about 20 hours on 4 NVIDIA A100 GPUs. The efficiency and smoothness introduced by the Mean Teacher self-training approach reduces the training iterations to 1/12 of that required by CutLER [38], highlighting the computation efficiency of our training strategy. Implementation details are included in Appendix J.
4.2 Evaluation Datasets and Metrics
We mainly conduct our experiments in a zero-shot manner on the validation sets of three benchmark datasets, namely Objects365 [27], LVIS [11], and SA-1B [18]. Given that self-supervised object detection methods, including HASSOD, do not utilize class labels as a form of supervision, we follow prior work [37-39] and evaluate these models as class-agnostic detectors, comparing them only against the bounding boxes and masks provided in the dataset annotations.