SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lingchen Meng $^{1,2}$  Xiyang Dai $^{3}$  Jianwei Yang $^{3}$  Dongdong Chen $^{3}$  Yinpeng Chen $^{3}$
Mengchen Liu<sup>3</sup> Yi-Ling Chen<sup>3</sup> Zuxuan Wu<sup>1,2†</sup> Lu Yuan<sup>3</sup> Yu-Gang Jiang<sup>1,2</sup>
<sup>1</sup>Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University
$^{2}$ Shanghai Collaborative Innovation Center of Intelligent Visual Computing
3Microsoft
Long-tailed object detection (LTOD) aims to handle the extreme data imbalance in real-world datasets, where many tail classes have scarce instances. One popular strategy is to explore extra data with image-level labels, yet it produces limited results due to (1) semantic ambiguity—an image-level label only captures a salient part of the image, ignoring the remaining rich semantics within the image; and (2) location sensitivity—the label highly depends on the locations and crops of the original image, which may change after data transformations like random cropping. To remedy this, we propose RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes. RichSem leverages rich semantics from images, which are then served as additional "soft supervision" for training detectors. Specifically, we add a semantic branch to our detector to learn these soft semantics and enhance feature representations for long-tailed object detection. The semantic branch is only used for training and is removed during inference. RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors. Our method achieves state-of-the-art performance without requiring complex training and testing procedures. Moreover, we show the effectiveness of our method on other long-tailed datasets with additional experiments. Code is available at https://github.com/MengLcool/RichSem.
Object detection for complex scenes has advanced significantly [10, 9, 40, 3, 65] thanks to large-scale datasets [5, 22, 32, 12, 24, 4, 42]. However, current deep models depend on relatively balanced large-scale datasets, where different classes have similar numbers of images and samples, to learn diverse semantics and enough locations. This limits their performance on real-world data, which are often long-tailed, i.e., only a few head classes have plenty of training samples while most classes have very few training samples, making detection more challenging and less effective for rare classes.
A simple and effective way to improve long-tailed object detection (LTOD) is to use extra data to increase the training samples for tail classes. However, collecting bounding box annotations, especially for rare categories, is costly and tedious. Therefore, previous studies resort to datasets with image-level labels to enrich the amount of samples for rare classes by exploring image-level semantics (as shown in Figure 1 (a)). While appealing, directly learning from such data to benefit detection is challenging since they lack bounding box annotations that are essential for object detection. To remedy this, many works [66, 25, 60, 63] focus on estimating bounding boxes for
(b)
Figure 1: (a) Category occurrences in the long-tail detection data (LVIS) and the extra classification data (ImageNet-21k). The green bars show the number of images that each category occurs in LVIS [12]; the blue dots show the number in ImagetNet-21k [5]. Evidently, the classification dataset exhibits a more evenly balanced distribution of occurrences. Moreover, the classification dataset offers a broader range of instances, thereby enhancing the diversity of instances for each category. (b) A sample from ImageNet [5]. This sample is annotated as "ball" while ignoring another main object "dog". After a random crop during training, the "ball" may be inaccurate on some crops. The class with the highest CLIP confidence score is shown above each crop.
objects in images. They typically consider image-level tags (in the form of one-hot labels) as the ground-truth classes and match with best estimated boxes as pseudo labels for training.
We argue that image-level labels are not well-suited for detection due to their semantics ambiguity and location sensitivity. On one hand, an image-level label can not sufficiently reflect all the semantics within the image. On the other hand, the label mainly focuses on the iconic object; thus, using a different crop might shift the semantics to a different object. Take Figure 1 (b) as an example: the top image is labeled as "ball" in ImageNet [5], which only depicts part of information in the image. The panel below in Figure 1 (b) shows that the provided image-level label is no longer accurate after data augmentations. Inspired by the recent success of visual-language contrastive models that align a large number of image-text pairs [37, 62, 11, 54, 13], we aim to leverage such models to extract rich semantics that are more informative than the image-level labels in classification datasets. However, naively converting one-shot labels in image classification tasks to a distribution of soft labels is still not optimal and accurate after data augmentation. As shown in Figure 1 (b), based on different locations (i.e., random crops) of the image, the semantics extracted by CLIP [37] incur more noise than the original image.
To address this, we introduce RichSem, a one-stage training framework that leverages additional image data to boost the detector through learning from rich semantics and coarse locations for long-tailed object detection. In particular, we treat a whole-image box as coarse locations (i.e., the coarse bounding box shares the same size as the image) and group multiple images together to build a mosaic. We then use CLIP to extract the semantics according to such coarse bounding boxes. The extracted semantics can serve as "soft labels" to enrich the amount of training data and are more robust to random cropping. We further introduce a new branch named as "semantic branch" to object detectors so that it can learn from the derived semantics during training. This allows the detector to fully leverage the rich semantics in classification datasets with minimal modifications. Once trained, this branch is discarded during inference, and RichSem can be used readily as a standard detector.
Our contributions are summarized as follows: (1) We point out that using image-level labels from extra classification datasets as supervision is challenging due to semantics ambiguity and location sensitivity, which can be alleviated with a distribution of semantics serving as "soft labels". We show that semantics provided by CLIP are not only rich and but also robust to locations, providing better guidance than original image-level labels. (2) We introduce a novel semantics learning framework named RichSem, which uses an additional branch to learn from rich semantics and coarse locations for long-tailed object detection without the need to compute pseudo labels. Once trained, the additional branch can be discarded during inference. (3) Our method demonstrates strong results on long-tailed datasets, e.g. LVIS, highlighting that it is a low-cost way to boost the detector performance for long-tailed object detection by only complementing extra classification data.
Long-tailed object detection (LTOD) has attracted more and more attention. The performance of rare categories is drastically low compared to frequent categories due to the unbalanced distribution and the lack of training samples. Existing works can broadly be divided into two directions: 1) one direction aims to improve the training scheme for balanced learning, including data re-sampling [12], loss re-weighting [46, 45, 51], data augmentations [8] and decoupled training [23, 21]; 2) Another direction leverages extra data to compensate for the data starvation [63, 60]. These methods often trust image-level labels to boost the classification capability. We find that it is far from optimal to naively treat image-level labels as golden labels and supervision for the classifier. Unlike existing methods, we study a new solution that leverages the rich semantics within the images from classification data with only coarse locations provided by regular augmentations. We leverage CLIP to provide semantics as soft targets on classification data to guide our detector to learn semantics explicitly. In this embarrassingly simple but effective way, we can better leverage classification data for LTOD.
Weakly-supervised object detection aims to train a detector using image-level labels without bounding boxes. Many studies [1, 47, 36] train a model using only image-level labels without any bounding box supervision. Another line of work [39, 60] takes the bounding boxes supervision with the whole-image labels together under a semi-supervised framework. Unlike prior works, we focus on mining rich semantics within the images instead of bounding box estimation. Thus, we no longer need to estimate precise bounding boxes on classification data.
Language supervision for object detection is a recent topic that aims to leverage linguistic semantics. Since language supervision has rich semantics, each category is related rather than independent in one-hot labels. Thanks to this property, recent works [6, 59, 57, 20, 35, 55] show benefits by pretraining backbones on vision-language tasks. With the rapid progress in contrastive language-image pre-training [37, 18], many recent approaches [28, 11, 62] apply large-scale pre-training for object detection. ViLD [11] and RegionCLIP [62] attempt to align the visual-semantic space of a pre-trained CLIP model for open-vocabulary detection. Similar to RegionCLIP and ViLD, our method leverages the visual-semantic space learned from pre-trained CLIP models. In contrast, our goal is to leverage object semantics to boost object classification, especially for rare categories. Since CLIPs are used to generate "soft labels", our backbone is free of CLIP initialization compared with RegionCLIP [62].
Knowledge distillation and soft label. Knowledge distillation (KD) [17] is a powerful tool to boost the student model with prior knowledge of the teacher model. Recent follow-ups extended and developed many variants, e.g., feature distillation [19, 41], hard distillation [50], contrastive distillation [48], etc. To improve the training efficiency of KD, Re-label [58] and FKD [44] use a strong teacher model to generate soft labels and store them for efficient image recognition training. Recently, many studies [11, 34, 62, 38] introduce knowledge from CLIP [37] into the KD framework to boost open-vocabulary object detection. Due to the strong semantics capturing capability of CLIP, those methods show encouraging performance on open-vocabulary detection. Similar to those works, our approach leverages the semantic knowledge of pre-trained CLIP models. In contrast, our goal is to boost the long-tailed detection, especially for the tailed classes, with the help of rich semantics of classification and detection datasets. Moreover, we introduce a simpler but more effective way to learn visual semantics with an extra semantic branch during training. Unlike [62, 11], our one-stage training scheme needs no fine-tuning after pre-training. Besides, our approach is more effective since there are no redundant pre-computed boxes [11] and boxes estimation [34, 38].
Given a detection dataset denoted as  $\mathcal{D}^{od} = \{(I^{od},\{(b^{od},c^{od})\})\}$ , where each image  $I^{od}$  is associated with bounding boxes  $b^{od}$  and class labels  $c^{od}$  respectively. Our goal is to leverage an additional classification dataset  $\mathcal{D}^{extra} = \{(I^{img},c^{img})\}$ , of which each image is only labeled with an image-level label  $c^{img}$ , to improve long-tailed object detection, particularly for rare classes.
In long-tailed object detection, previous approaches typically rely on image-level labels and compute pseudo boxes for corresponding objects. On one hand, the semantic information provided by such labels is limited, (i.e., one one-hot label per image). On the other hand, augmentation strategies like randomized cropping might generate regions that do not contain the annotated class, thus making the provided labels no longer accurate. Motivated by the fact that CLIP has strong capabilities of
capturing visual semantics conditioned on only coarse locations [62], we build upon CLIP to explore image classification datasets to guide the training of object detectors. Below, we first introduce how to obtain rich semantics from CLIP in Section 3.1, and the resulting semantics are then used to guide the training of our detector, as will be described in Section 3.2. Then we describe how we unify the training objective and extend RichSem to different types of extra datasets in Section 3.3.
Figure 2: Our RichSem framework.  $\mathcal{V}_t$  and  $\mathcal{L}_t$  indicate vision encoder and text encoder of the CLIP model. We leverage rich semantics in extra data with image-level labels by pre-trained CLIP. Then semantics behave like "soft labels" for our proposed semantic branch ( $H_{soft}$  followed by Contrast). Blue parts indicate the components for traditional object detection and green parts indicate those for semantics learning.
3.1 Image and Object Semantics
We aim to use image-level labels in classification datasets together with detection datasets to improve long-tailed object detection. This requires processing these two types of data within the same framework. As a result, we treat images from detection and classification datasets equally and we not only derive semantic information at the image-level for classification datasets but also at the object-level for detection datasets, as will be elaborated below.
Image-level semantics. We denote the pre-trained CLIP visual encoder and the language encoder as  $\mathcal{V}_t$  and  $\mathcal{L}_t$ , respectively. Consider an image  $I$  after random cropping, its semantics  $s$  can be obtained by computing the similarity between its visual features  $f^{img}$  extracted by the visual extractor and the linguistic categories features  $f^{cats}$  produced by the language encoder. Formally, we compute the visual and linguistic features as follows:
$$
f ^ {i m g} = \mathcal {V} _ {t} (I ^ {i m g}); \quad f ^ {c a t s} = \mathcal {L} _ {t} (\mathcal {P} (\mathcal {C} ^ {o d}))
$$
$$
s ^ {i m g} = \operatorname {C o n t r a s t} (f ^ {i m g}, f ^ {c a t s}).
$$
Here,  $\mathcal{P}$  is the prompt engineering function to convert class names into prompts; Contrast is the contrasting paradigm [37] to calculate the similarity between two features, and  $s^{img}$  contains semantics information in which each element indicates the likelihood for the corresponding class. Following [37, 62], we use a set of prompts to convert the categories in the target vocabulary  $\mathcal{C}^{od}$  into sentences then use the CLIP text encoder  $\mathcal{L}_t$  to obtain the linguistic features of categories in  $\mathcal{C}^{od}$ .
Object-level semantics. As mentioned above, to unify the training process with classification and detection data, we also obtain object-level semantics from CLIP as well. To this end, we obtain an object-level representation by pooling from image-level features [9, 15] rather than cropping in the original images, following [62]. More formally, object-level semantics are obtained as:
$$
f ^ {t} = \operatorname {P o o l e r} \left(\mathcal {V} _ {t} (I), b\right)
$$
$$
s ^ {t} = \operatorname {C o n t r a s t} \left(f ^ {t}, f ^ {c a t s}\right) \tag {2}
$$
1Please refer to our supplementary material for the verification of coarse location
where Pooler is RoIAlign [15] that pools region features according to their locations from the entire feature maps, and  $b$  is the location of the object, which could be ground-truth, predicted, or predefined whole-image bounding boxes.
It is worth pointing out with such a formula both detection and classification datasets can now be unified, as objects in detection datasets are essentially regions of images, while images from classification datasets are cropped regions from original images. Consequently, the  $b$  in Equation (2) indicate tight ground-truth bounding boxes  $b^{od}$  for objects in detection datasets, while  $b$  corresponds to the entire image, i.e., coarse whole-image boxes  $b^{img} = (0, 0, h, w)$ , for samples from classification datasets, where  $h, w$  is the height and width of the augmented image.
Furthermore, unlike previous approaches that use fixed and static semantics provided by image-level labels, we obtain image-level and object-level semantics which dynamically change conditioned on different locations ( i.e., bounding boxes) in an online fashion. This is particularly useful when location information is not accurate—the location could be a very loose bounding box for the object of interest or even contains part of the objects. As a result, the rich and location-robust semantics can be used to guide the detector.
3.2 Semantics as Soft Labels
Now that we obtain both image-level and object-level semantics with CLIP, the main challenge is how to make our detector learn from them effectively. Traditional detection models [40, 31, 2, 3, 65] usually obtain object features from whole-image features according to locations [9, 40, 31, 49] or cross-attention [3, 65]. Then object features are refined with a classification branch to classify the region into predefined categories, and a location branch to compute the bounding box, separately. To unify the notations, we denote the object features as  $O$ , which are fed to different branches to obtain features tailored for each sub-task to generate final outputs with a projection layer. More formally,
$$
f ^ {l o c}, f ^ {c l s} = H _ {l o c} (O), H _ {c l s} (O)
$$
$$
\begin{array}{l} j, j = \mathbf {H} _ {l o c} (\odot), \mathbf {H} _ {c l s} (\odot) \\ o ^ {l o c}, o ^ {c l s} = W _ {l o c} (f ^ {l o c}), W _ {c l s} (f ^ {c l s}), \end{array} \tag {3}
$$
where  $H$  denotes the sub-task branch for feature refinement, specifically  $H_{loc}$  is for localization and  $H_{cls}$  is for classification;  $W_{t}$  denotes the projection layer to produce final results;  $f_{t}$ , and  $o_{t}$  denotes the features and outputs for the corresponding task, i.e.,  $t \in \{loc,cls\}$ .
Now we discuss how to use the obtained semantics to guide the training of detectors. As  $o^{cls}$  are logits that are generally used for classification, a straightforward way is, on top of a cross-entropy loss, to use  $o^{cls}$  to predict the object semantics  $s^t$  in a similar spirit to knowledge distillation [17, 41]. However, this makes the training of detectors challenging as the cross-entropy loss and distillation loss have conflicting purposes: a cross-entropy loss is generally optimized to produce hard prediction results while the distillation loss aims to borrow knowledge from a distribution of semantic scores that serve as "soft labels". As a result, jointly performing hard predictions and soft distillations conditioned on  $o^{cls}$  produce unsatisfactory results, as will be shown empirically.
To mitigate the challenge, we introduce an additional branch named as "semantic branch", independent of the classification and localization branches in current detectors, to learn the soft semantics obtained by CLIP. The semantic branch also includes a feature refinement branch  $H_{soft}$  then feed the refined feature  $f^{soft}$  to Contrast function to obtain the semantic prediction  $o^{soft}$ . Such a strategy ensures that semantics from image classification datasets can be explored without interfering with the original training process of detectors. This is essentially using semantics as soft teachers to implicitly refine object features for detection. Formally, to train the semantic branch, we use a KL divergence loss as:
$$
o ^ {s o f t} = \operatorname {C o n t r a s t} \left(f ^ {s o f t}, f ^ {c a t}\right), \quad f ^ {s o f t} = H _ {s o f t} (O)
$$
$$
L _ {\text {s o f t}} = \frac {1}{N} \sum_ {i = 1} ^ {N} L _ {K L} \left(o _ {i} ^ {\text {s o f t}}, s _ {i} ^ {t}\right) \tag {4}
$$
where  $o_i^{soft}$  is the semantics prediction of the  $i$ -th object feature, while  $s_i^t$  is the corresponding semantic target;  $N$  is the number of matched proposals/ queries during training. It is worth noting that we only use the semantic branch for distilling semantics into the detector during training. And once trained, the semantic branch is no longer needed for inference.
3.3 Unified Objective Functions
We aim to incorporate semantic learning from classification datasets into an end-to-end learning scheme instead of fine-tuning after pre-training [62], without redundant pre-computed boxes [11] and box estimation [34, 38]. Thus, we unify the objective functions and treat datasets of classification and detection equally in a unified way.
In our training scheme, we use a unified classification loss  $L_{unicls}$ , the combination of a hard classification loss and a soft semantics loss, to boost the capability of object classification. The hard classification loss  $L_{cls}$  best works when bounding boxes and class labels are available and sufficient for training. In contrast, the soft semantic learning loss  $L_{soft}$  works well for those categories with few samples, requiring only coarse locations.
$$
L = \lambda_ {l o c} \cdot L _ {l o c} + \underbrace {\lambda_ {c l s} \cdot L _ {c l s} + \lambda_ {s o f t} \cdot L _ {s o f t}} _ {L _ {u n i c l s}} \tag {5}
$$
where  $\lambda_{loc}$ ,  $\lambda_{cls}$  and  $\lambda_{soft}$  means the weight of each loss.
Furthermore, the vocabularies of target detection datasets and extra classification datasets often differ. To handle the taxonomy difference between detection and classification datasets, we further unify the supervision for hard classification as follows.
Handling taxonomy differences. To leverage image-level labels in extra data, we need to map the vocabulary of extra data  $\mathcal{C}^{extra}$  to the vocabulary of our target detection dataset  $\mathcal{C}^{od}$ , which is a function  $\mathcal{M}: \mathcal{C}^{extra} \to \mathcal{C}^{od}$ . One could derive better mapping functions  $\mathcal{M}$ , yet this is orthogonal to our current direction. For convenience, we can use a manual label mapper or leverage the CLIP text encoder to map the class according to their category semantics automatically.
$$
c ^ {o d} = \mathcal {M} \left(c ^ {i m g}\right) \tag {6}
$$
Handling data without any labels. We also consider a broader case, in which even image-level labels are not available. Inspired by DeiT [50], we treat the class of highest logits in each semantics target as the label for classification. We also filter  $c^{hard}$  with a threshold to filter out those images relevant to the vocabulary of the target set.
$$
c ^ {h a r d} = \left\{ \begin{array}{l l} \arg \max  \left(s ^ {t}\right) & \text {i f c o n f} > t h \\ \varnothing & \text {e l s e} \end{array} \right. \tag {7}
$$
where  $c^{hard}$  is the generated class label for hard classification.
We conduct experiments and analysis on the task of long-tailed object detection. We mainly evaluate our method on LVIS [12] val 1.0 for our main experiments and ablations. We also conduct experiments on other datasets of long-tail distribution to further prove the effectiveness. We use DINO [61], a advanced DETR-based detector due to the training efficiency and high performance.
4.1 Datasets and evaluation metrics.
Long-tail detection dataset. We mainly conduct experiments on LVIS [12], which contains 1203 classes with  $\sim 100\mathrm{K}$  images. The classes are divided into rare, common, and frequent groups based on the number of training images. The category distribution is extremely long-tailed since instances in rare categories are usually less than 10. Moreover, we experiment on other datasets, e.g. Visual Genome [22] and OpenImages [24], please refer to our supplementary material.