SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Extra data. We mainly use ImageNet-21k [5] as additional classification data. ImageNet-21k consists of  $\sim 14\mathrm{M}$  images for 21K classes, and there are 997 classes that overlap with the vocabulary in LVIS. We follow the label mapping used in Detic [63] and denote the subset of overlapped classes as ImageNet-LVIS, which contains  $\sim 1.5\mathrm{M}$  images with 997 LVIS classes. For the full set, we treat the data as unlabeled images and leverage our learning scheme to learn from it. We also treat the
Method	Backbone	Schedule	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_r \)	\( AP_c \)	\( AP_f \)
DINO*[61]	R50 [16]	1×	28.8	38.4	30.3	18.2	26.5	36.1
MosaicOS† [60]	R50 [16]	1×	25.0	40.8	26.5	20.2	23.9	28.3
Detic-DDETR [65]	R50 [16]	4×	31.7	-	-	21.4	30.7	37.5
Detic-DDETR† [63, 65]	R50 [16]	4×	32.5	-	-	26.2	31.3	36.6
Detic-CenterNet2 [63]	R50★ [16]	4×	35.3	48.7	37.2	28.2	33.8	40.0
Detic-CenterNet2† [63]	R50★ [16]	4×	36.8	50.7	38.6	31.4	36.0	40.1
RichSem (Ours)	R50 [16]	1×	32.2	42.3	33.9	24.1	29.9	38.3
RichSem (Ours)	R50 [16]	2×	34.9	45.5	36.6	26.4	32.5	41.3
RichSem (Ours)	R50 [16]	3×	35.1	45.8	36.8	26.0	32.6	41.8
RichSem‡ (Ours)	R50 [16]	1×	35.0↑2.8	46.0	36.7	30.4↑6.3	33.1	39.0
RichSem‡ (Ours)	R50 [16]	2×	37.1↑2.2	48.2	39.0	29.9↑3.5	35.6	42.0
RichSem‡ (Ours)	R50★ [16]	2×	40.1↑5.2	51.9	42.3	36.2↑11.8	38.2	44.0
RichSem (Ours)	Swin-T [33]	1×	34.9	45.5	36.9	26.0	32.6	41.3
RichSem‡ (Ours)	Swin-T [33]	1×	38.3↑3.4	49.8	40.4	34.1↑8.1	36.3	42.3
RichSem (Ours)	Swin-T [33]	2×	38.8	49.9	41.0	30.8	36.4	45.0
RichSem‡ (Ours)	Swin-T [33]	2×	41.6↑2.8	53.3	43.8	37.3↑6.5	39.7	45.5
Detic-CenterNet2 [63]	Swin-B [33]	4×	45.4	59.9	47.9	39.9	44.5	48.9
Detic-CenterNet2†[63]	Swin-B [33]	4×	46.9	62.2	49.4	45.8	45.5	49.0
RichSem (Ours)	Swin-B [33]	2×	46.4	59.2	48.9	38.5	45.1	51.3
RichSem‡ (Ours)	Swin-B [33]	2×	48.2↑1.8	61.6	51.0	46.5↑8.0	46.5	51.0
ViTDet [29]	ViT-L-MAE [14]	~8×	51.2/ 49.2	-	-	-	-	-
ViTDet [29]	ViT-H-MAE [14]	~8×	53.4/ 51.5	-	-	-	-	-
RichSem (Ours)	Swin-L [33]	1×	47.0	59.9	49.6	41.2	45.9	50.7
RichSem‡ (Ours)	Swin-L [33]	1×	49.8↑2.8	63.7	52.5	48.6↑7.4	49.7	50.5
RichSem (Ours)	Swin-L [33]	2×	49.7	62.9	52.4	42.8	49.2	53.4
RichSem‡ (Ours)	Swin-L [33]	2×	52.0↑2.3	65.7	54.8	50.2↑7.4	51.5	53.3
RichSem (Ours)	Focal-L [56]	1×	49.5	62.5	52.2	47.3	47.7	52.4
RichSem‡ (Ours)	Focal-L [56]	1×	51.5↑2.0	65.2	54.4	52.3↑5.0	50.7	52.0
RichSem (Ours)	Focal-L [56]	3×	51.4	64.8	54.2	47.6	49.8	54.9
RichSem‡ (Ours)	Focal-L [56]	3×	53.6↑2.2	67.2	56.7	52.8↑5.2	52.6	55.2
Table 1: Results on LVIS val v1.0. * indicates that we train the model on LVIS using their official hyperparameters best trained on COCO; R50 indicates ResNet50 [16] pre-trained on ImageNet-1k, while R50* indicates pre-trained on ImageNet-21k; RichSem (Ours) indicates that we train our baseline model under the original object detection dataset with traditional loss; RichSem  $\ddagger$  (Ours) indicates using additional extra data under our training scheme. Besides,  $\dagger$  indicates that the model is trained with extra data. In this table, our models use the subset of ImageNet-21k, of which categories overlap with LVIS vocabulary as extra dataset. The AP of ViTDet includes the AP reported paper (the former) and the AP reported in their official repo (the later).
full set as unlabeled dataset ignoring the image-level labels, denoted as INet-Unl. Furthermore, we explore an additional detection dataset Object365 [42] and an image-text pair dataset CC3M [43].
Evaluation metric. We report box AP on LVIS with the LVIS official evaluator, which also contains  $\mathrm{AP}_r$ ,  $\mathrm{AP}_c$ ,  $\mathrm{AP}_f$  for rare, common and frequent class, respectively. We mainly focus on the performance gain on overall AP and AP of rare categories.
Baseline model. We take DINO [61], a powerful DETR-based detector with advanced query denoising [27] as our baseline. Differently, we convert the object classification into vision-language contrasting using the text features of categories.
Mixed dataset training. For experiments with extra data, we combine both detection datasets and extra datasets together to train our detector. We sample images from detection and classification datasets in a 1:1 ratio regardless of the original size of each dataset. Images in the same batch are sampled from the same dataset for high training efficiency. For mixed dataset training, we denote the schedule as the number of iterations on the target dataset for fair comparisons.
Training details. We adopt PyTorch for implementation and use  $8 \times \mathrm{V}100$  GPUs. We set the initial learning rate as 1e-4 and multiply 0.1 at the 11-th, 20-th and 30-th epoch for  $1 \times$ ,  $2 \times$  and  $3 \times$ , respectively, and set  $\lambda_{soft} = 0.5$  for the soft semantics learning loss. Following [63], we use a federated loss [64] and repeat factor sampling [12] for LVIS; we use category aware sampling for OpenImages. We randomly resize an input image with its shorter side between 480 and 800, limiting
the longer size below 1333. Unlike training large models with larger scale images [29, 61], we use the same recipe for all models without any tricks or test time augmentation (TTA). For training with extra data, we use CLIP-RN50 to extract the semantic guidance for most models, while we use CLIP-RN50×16 [37] for Swin-L [33] and Focal-L [56] for better performance.
Results on LVIS. Tab. 1 shows the result on LVIS. As shown in the table, detectors trained with our framework outperforms those trained with regular training recipes, especially for rare categories.
When using a ResNet50 [16] as backbone networks, our baseline model is fully converged under  $3\times$  schedule. Our RichSem under  $2\times$  still outperforms the fully converged baseline, which demonstrates the effectiveness of our proposed method. Moreover, the experiments with Swin [33] backbones show consistent gains. Our model with a Swin-L as backbone achieves 52.0 AP and  $50.2\mathrm{AP}_r$  ,which outperforms the previous SoTA [29] by a large margin only under 1/4 training schedule. Overall, our RichSem makes the detector better on tailed categories and achieves significant gains on the corresponding AP, e.g. rare and common categories. Moreover, with better backbones, our RichSem achieves even balanced perfor
mance on rare, common and frequent categories. Notably, with a much smaller model size and standard data augmentations for training and testing, our method rivals the ViTDet [29] that uses huge-sized model and sophisticated training recipes. We also compare compare the models with Faster R-CNN [40] as detector shown in Table 2. We further compare with large-scale vision foundation models [7, 53], please refer to our supplementary material.
Method	AP	APr	APc	APf
Faster R-CNN [40]	24.1	14.7	22.2	30.5
EQL-v2 [45]	25.5	17.5	23.9	31.2
BAGS [30]	26.0	17.2	24.9	31.1
Seesaw Loss [51]	26.4	17.5	25.3	31.5
EFL [26]	27.5	20.2	26.1	32.4
MosaicOS ‡ [60]	23.9	15.5	22.4	29.3
CLIS ‡	29.2	24.4	28.6	31.9
RichSem ‡ (Ours)	30.6	27.6	29.7	32.9
Table 2: Results on LVIS val v1.0 with Faster R-CNN as detector. All models use R50 as backbone. † indicates using extra classification data.
Comparisons with CLIP initialization. We replace the ResNet50 backbone with pre-trained CLIP visual encoder to study whether it can lead to balanced learning. As shown in Table 3a, although pre-trained CLIP has a strong ability to capture visual semantics, the detection performance even drops after finetuning on LVIS. The low  $\mathrm{AP}_r$  of CLIP backbone indicates that it is still biased towards the long-tailed distribution of downstream datasets. Therefore, using a frozen pre-trained CLIP to extract visual semantics as extra "soft supervision" can keep the generalization capability than building upon CLIP as initialization.
Effectiveness of the semantics branch. We study the importance of the proposed semantics branch by sharing the parameters of two heads instead of using two independent heads. As shown in Table 3b, when two heads share the parameters, the performance drops drastically and is even worse than the baseline model. This suggests that the objective of semantics learning differs from object classification. Therefore, semantics should be learned independently with the semantics branch.
Effectiveness of rich semantics of classification data. We conduct experiments using different types of extra data. Initially, we utilized Object365 [42], a detection dataset, and utilized its box annotations irrespective of the class labels. However, the performance gain was limited, as shown in Table 3c. This further verifies the assumption that the localization capability is not the primary bottleneck for long-tailed object detection [11]. Next, we experiment with ImageNet [5] and CC3M [43], two datasets with whole-image labels. We treat these datasets as unlabeled data and generate pseudo labels using Equation (7), denoted as INet-Unl and CC3M-Unl, respectively. As shown in Table 3c, utilizing these two datasets as extra data significantly outperformed the baseline, particularly for the rare categories. Furthermore, we employed ImageNet-LVIS, taxonomy mapped using Equation (6), as extra data, which yields the best performance. Overall, the experiments demonstrate that long-tailed object detection is primarily affected by object classification. Consequently, introducing rich semantics within classification data effectively alleviates this limitation. It is worth noting that this showcases our potential for further extension to unlabeled data.
Method	AP	APr
baseline	32.2	24.1
CLIP init backbone	30.6↓1.6	22.2↓1.9
RichSem ÷	35.0↑2.8	30.4↑6.3
Extra branch	AP	APr	APc	APf
baseline	32.2	24.1	29.9	38.3
X	31.1↓1.1	24.6↑0.5	28.3↓1.6	37.2↓1.1
✓	35.0↑2.8	30.4↑6.3	33.1↑3.2	39.0↑0.7
(a) Compared with CLIP initialized baseline.
Extral	AP	APr
None	32.2	24.1
O365-Box	33.0↑0.8	24.8↑0.7
CC3M-Unl	34.0↑1.8	28.7↑4.6
INet-Unl	34.7↑2.5	28.6↑4.5
INet-LVIS	35.0↑2.8	30.4↑6.3
CLIP models	AP	APr
None	32.2	24.1
RN50	35.0↑2.8	30.4↑6.3
RN50×4	36.0↑3.8	33.0↑8.9
RN50×16	36.2↑4.0	31.9↑7.8
(b) Ablations on the semantic branch.
Extra data	Lsoft	AP	APr
None	32.2	24.1
INET-LVIS	X	33.8↑1.6	26.9↑2.8
✓	35.0↑2.8	30.4↑6.3
INET-Unl	X	32.9↑0.7	23.7↓0.4
✓	34.7↑2.5	28.6↑4.5
(c) Different extra datasets.
(d) Soft semantics provided by different CLIP models.
Dbackbone	Od	AP	APr	APc	APf
IN-1K	LVIS	32.2	24.1	29.9	38.3
IN-21K	LVIS	35.7	25.9	35.0	40.7
IN-1K	LVIS+IN-21K	35.0	30.4	33.1	39.0
IN-21K	LVIS+IN-21K	37.5	32.4	36.0	41.5
(e) Soft semantics loss  $L_{soft}$ .
Method	AP	APr	APc	APf
w/o Dextra	32.2	24.1	29.9	38.3
+Lsoft	33.6	28.6↑4.5	32.4	37.2
+Lloc	35.0	30.4↑1.8	33.1	39.0
(f) Ablations on backbone pre-training data and downstream detection data.
(g)  $L_{soft}$  and  $L_{loc}$  on extra data.
λsoft	AP	APr	APc	APf	AP	APr	APc	APf
# Only on LVIS	# with Image-LVIS
0	32.2	24.1	29.9	38.3	33.8	26.9↑2.8	31.5↑1.6	39.4↑1.1
0.2	32.4	23.8↓0.3	30.5↑0.6	38.5↑0.2	35.0	29.0↑4.9	33.3↑3.4	39.4↑1.1
0.5	32.4	25.0↑0.9	30.5↑0.6	37.7↓0.6	35.0	30.4↑6.3	33.1↑3.2	39.0↑0.7
1.0	32.1	27.0↑2.9	29.8↓0.1	36.9↓1.4	34.8	29.5↑5.4	33.4↑3.5	38.7↑0.4
(h) Ablations on the weight of soft semantics loss.
Table 3: RichSem ablations. All ablations are performed with RN50 as backbone under  $1\times$  schedule.
Stronger semantics lead to better detectors. We study the impact of CLIP with different backbones. The results are shown in Table 3d. With the capability increased, the model achieves better performance on overall AP. Besides, built upon a  $\mathrm{RN}50\times 16$ , our detector obtains significant improvements on all metrics.
Semantic learning is effective to leverage extra data. Due to the data overlap between backbone pretraining and detector training, we conduct an ablation on them to further demonstrate the effectiveness of our method. As shown in the Table 3f, pre-training on large-scale data (ImageNet-21k) can provide strong perception capability for the downstream detection task, with overall performance gain. However, the performance on rare categories is still relatively low, indicating that this approach does not well alleviate the long-tail effects in detection. In contrast, our method is more effective than pre-training to handle long-tailed detection, especially for the tail categories. Notably, our approach is still effective with strong pre-trained backbones, further improving performance on long-tailed object detection.
Semantics as soft targets leads to balanced learning. The effectiveness of soft labels for low-shot categories can be reaffirmed in Table 3e: when using extra data, detectors trained with the soft semantics learning loss achieve better performance than those trained without the loss, especially for rare categories and unlabeled extra data. Besides, our semantic learning on extra classification boosts detection and classification at the same time thanks to our unified objective functions ( Section 3.3). As shown in Table 3g, both rich semantics and coarse locations play significant roles in boosting long-tail object detection. We further study the impact of soft classification objectives. We first experiment without using extra data. As shown in the Table 3h, with distillation weight set to 1, the performance of rare categories increases by  $\sim 3\mathrm{AP}$ , while the results of frequent categories decrease by 1.4. This suggests that semantics learning benefits those low-shot categories rather than overfitting frequent categories. As the distillation weight decreases, the performance of rare classes also drops but the results of frequent categories improve. After exploring additional data, the performance is no longer sensitive to the distillation weight. Thanks to the additional data, semantics learning leads to
Figure 3: Qualitative results of our RichSem. We show the images containing rare categories from LVIS val 1.0. We show rare categories in red and show others in black. Best viewed on sceen.
Figure 4: Feature visualization between baseline and RichSem. We randomly sample two classes each from the categories of rare, common, and frequent for visualization. Our RichSem shows well-clustered result across different categories.
consistent performance gains for all categories. We further visualize the object features of our method. Specifically, we normalize the features and employ Gaussian Kernel Density Estimation (KDE) in  $\mathbb{R}^2$ , following [52] to compare the distribution of object features across categories. As shown in the Figure 4, the visualization indeed shows a clear distinction between the baseline and our RichSem. Regarding the baseline, the distribution of object features lacks differentiation, often resulting in overlapping patterns among categories, especially between rare and frequent categories. In contrast, in RichSem, features belonging to each category, even rare categories, are well-clustered. This clear intra-class and inter-class distribution indicate that our approach effectively enhances the region classification capability of diverse categories. Therefore, our method effectively leads balanced learning among classes with varying frequencies.
We presented RichSem, a simple but effective way, to leverage extra data by learning rich semantics and coarse locations to boost long-tailed object detection, alleviating the semantics insufficiency and location sensitivity caused by taking image-level labels as supervision. Through extensive experiments, we demonstrate our approach achieves state-of-the-art performance, without requiring complex training and testing procedures. One possible limitation is that we treat the detection data and classification data equally and use the same unified classification loss, which may be sub-optimal for the categories with sufficient samples.
For future work, we believe our framework can be extended to semi-supervised object detection (SSOD) where golden labels are not annotated on the unlabeled data, since our method is found to be robust when annotations are partially given. Moreover, the soft semantics learning for classification data can be naturally applied for open-vocabulary and multi-dataset detection tasks.
Acknowledgement This project was supported by National Key R&D Program of China (No. 2021ZD0112805) and National Natural Science Foundation of China (No. 62102092).
[51] J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin. Seesaw loss for long-tailed instance segmentation. In CVPR, 2021.
[52] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[53] W. Wang, J. Dai, Z. Chen, Z. Huang, Z. Li, X. Zhu, X. Hu, T. Lu, L. Lu, H. Li, et al. Interimage: Exploring large-scale vision foundation models with deformable convolutions. In CPVR, 2023.
[54] Z. Wang, N. Codella, Y.-C. Chen, L. Zhou, X. Dai, B. Xiao, J. Yang, H. You, K.-W. Chang, S.-f. Chang, et al. Multimodal adaptive distillation for leveraging unimodal encoders for vision-language tasks. arXiv preprint arXiv:2204.10496, 2022.
[55] Z. Weng, X. Yang, A. Li, Z. Wu, and Y.-G. Jiang. Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. In ICML, 2023.
[56] J. Yang, C. Li, X. Dai, and J. Gao. Focal modulation networks. In NeurIPS, 2022.
[57] K. Ye, M. Zhang, A. Kovashka, W. Li, D. Qin, and J. Berent. Cap2det: Learning to amplify weak caption supervision for object detection. In CVPR, 2019.
[58] S. Yun, S. J. Oh, B. Heo, D. Han, J. Choe, and S. Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In CVPR, 2021.
[59] A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
[60] C. Zhang, T.-Y. Pan, Y. Li, H. Hu, D. Xuan, S. Changpinyo, B. Gong, and W.-L. Chao. Mosaicos: a simple and effective use of object-centric images for long-tailed object detection. In ICCV, 2021.
[61] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. Ni, and H.-Y. Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2022.
[62] Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, 2022.
[63] X. Zhou, R. Girdhar, A. Joulin, P. Krahenbuhl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022.
[64] X. Zhou, V. Koltun, and P. Krahenbuhl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461, 2021.
[65] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[66] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. Le. Rethinking pre-training and self-training. In NeurIPS, 2020.