SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation. We use mean average precision (mAP) on LVIS [10] as our main detection metric, where  $\mathrm{mAP}_{\mathrm{rare}}$  indicates open-vocabulary performance on unseen classes. To measure generalization on diverse real-world tasks, we evaluate zero-shot performance on the "Object Detection in the Wild" (ODinW) benchmark [21]. ODinW is a suite of datasets covering a wide range of domains. We report the average mAP on the subset of 13 ODinW datasets introduced in [22] and provide performance on individual datasets in Appendix A.9.2. To avoid leakage of evaluation data into the training set, WebLI was filtered to remove images similar to those in the train, validation, and test splits of 68 common computer vision datasets, including COCO/LVIS, Objects365, and Visual Genome, but not the ODinW datasets (see [4] for details).
Figure 3: Impact of pseudo-annotation filtering by detection confidence on self-training effectiveness. Pseudo-labels (N-gram label space) were filtered using different confidence thresholds. Number of remaining images for each threshold: 0.1: 5B, 0.3: 2B, 0.5: 782M, 0.7: 224M. OWLv2-B/16 detectors were self-trained on the filtered pseudo-annotations and fine-tuned on  $\mathrm{LVIS}_{\mathrm{base}}$ . Each point represents a different fine-tuning run. "Examples seen" refers to the number of images after creating mosaics; the total number of raw images seen is  $13.2\times$  that number (Section 3.2).
We compare our best models to the literature in Table 1. We broadly include state-of-the-art open-vocabulary detectors in the comparison. Our self-training approach, using only machine-generated pseudo-annotation queries, improves over previous methods even without fine-tuning (Table 1, OWL-ST, rows 11–13). Our OWL-ST B/16 model (row 11) achieves  $29.6\%$  LVIS  $\mathrm{mAP}_{\mathrm{rare}}$ , 9 points more than the equivalent OWL-ViT model (row 2). Our largest model, G/14 (row 13), reaches  $37.5\%$ $\mathrm{mAP}_{\mathrm{rare}}$ , 4.7 points better than the next-best model from the literature (F-VLM R50x64, row 8). Interestingly, after self-training, our models perform better on LVIS  $\mathrm{mAP}_{\mathrm{rare}}$  than  $\mathrm{mAP}_{\mathrm{all}}$  (which includes frequent and common classes). We speculate that this may be because weak Web-data supervision may be better for specific terms than general terms: Image/text pairs involving unusual objects (such as  $\mathrm{LVIS}_{\mathrm{rare}}$  categories) may be more likely to be specifically about these objects, whereas common terms like "person" or "car" may occur often without being related to the image.
Fine-tuning on  $\mathrm{LVIS}_{\mathrm{base}}$  provides additional significant improvements, even on  $\mathrm{mAP}_{\mathrm{rare}}$  (OWL-ST+FT, rows 14-16). Our best model, which has only seen machine-generated queries during self-training, reaches  $47.2\%$  LVIS  $\mathrm{mAP}_{\mathrm{rare}}$  after fine-tuning, a 14.4-point improvement over the next best model (F-VLM R50x64, row 8).
Including a human-curated list of common object classes as pseudo-annotation queries can further improve the results on LVIS (rows 20–21), but this approach is not fully open-vocabulary since the model sees a curated label space, including the LVIS classes, at training time. While the benefit of the curated label space is significant for our smallest model, is minor on  $\mathrm{mAP}_{\mathrm{rare}}$  for the larger L/14 model (compare rows 15 and 21).
To measure more general open-world performance, Table 1 also includes zero-shot results on ODinW13 [21], a suite of "in the wild" datasets. Performance on ODinW is best right after self-training and is reduced by fine-tuning on  $\mathrm{LVIS}_{\mathrm{base}}$ . We discuss this further in Section 4.6. We also fine-tuned on COCO, where our B/16 and L/14 models reach  $54.3\%$  and  $56.0\%$  COCO mAP, respectively. OWLv2 therefore matches the performance of ViTDet with a Cascade Mask-RCNN head [23], despite using a simpler head architecture. Further results and examples in Appendix A.9.
4.3 Pseudo-Annotation Label Space
Figure 2 takes a closer look at the impact of the pseudo-annotation label space on performance after fine-tuning. Performance on fine-tuned classes (mAP_{frequent}; left plot) is highest if the pseudo-annotation label space included these classes (blue circles). Therefore, if the target label space is known ahead of time, pseudo-labeling on that space leads to the best results.
However, performance on unseen classes  $\left(\mathrm{mAP}_{\mathrm{rare}}\right)$  and "In the Wild" datasets is much better if the pseudo-labeling included diverse queries that were machine-generated from the image-associated text (orange squares and green diamonds). A mixture of human and machine-generated label spaces
Figure 4: Scaling of detection performance with model size and training compute. Models show classic scaling behavior [42]: Performance increases monotonically with training compute, with larger models being necessary to benefit from larger amounts of compute/data. Models were self-trained on N-gram pseudo-annotations and fine-tuned on LVIS<sub>base</sub>.
performs well in all settings, but does not significantly outperform the purely machine-generated label space on the "In the Wild" datasets. These results suggest that a human-curated label space can help if the target label space is known, but that strong in-the-wild generalization is driven by the weakly supervised machine-generated label space. Our results also show that a simple N-grams approach is sufficient to leverage the weak supervision and outperforms more complex methods (Table 1).
4.4 Filtering of Pseudo-Annotations
Besides the label space, a second important decision in self-training is the filtering of pseudo-annotations. We filter based on the detection confidence score of the annotator and vary the score threshold in Figure 3. For confidence-based filtering, a bias-variance trade-off exists between including only high-confidence pseudo-annotations but inheriting the annotator's biases, or lowering the bias but increasing the noise by including lower-confidence pseudo-annotations. Many prior works err on the side of high bias and low variance, applying high confidence thresholds [35] or including only the single highest-confidence detection for an image [47, 1]. In our setting, we find that including all pseudo-annotations that pass a moderate threshold of 0.3 works well, while strict thresholds lead to poor results (Figure 3). As training continues for longer than what was possible for Figure 3, we suspect that lower thresholds may scale better. Therefore, for our main results, we chose to include all annotations above 0.1, but only kept images with at least one annotation above 0.3.
4.5 Scaling
The use of abundant Web image-text data with little filtering means that our self-training dataset is large (approximately 2B images). We can therefore study detection training scaling in the same regime as prior work on classification (Figure 4; models see each image at most once for these experiments). We make several noteworthy observations:
1. Self-training is beneficial already at moderate compute budgets, less than that of the annotator.
2. Models show similar scaling behavior for detection as for classification [42]: Both overall performance and the size of the Pareto-optimal model increase with compute/data size.
3. As we move further out of distribution, the amount of compute at which L/14 overtakes B/16 increases. In other words, for in-the-wild performance, at most compute budgets, it may be better to train a smaller model for longer than a larger model for shorter.
These results suggest that self-training on Web data is further scalable as an approach for improving open-vocabulary localization models without the need for further human annotations. The large datasets also makes it possible to scale model size. We trained a G/14 model, which has  $5.2 \times$  the number of parameters and  $4.3 \times$  the inference FLOPs of our L/14 model. To our knowledge, this is the largest open-vocabulary detection model to date. Since the G/14 model uses a different backbone than our other models (SigLIP [43] instead of CLIP [30]), we do not include it in Figure 4, but show in Table 1 that it is currently the best-performing model on zero-shot LVIS, with  $47.2\%$ $\mathrm{mAP}_{\mathrm{rare}}$ .
Figure 5: Trade-off between fine-tuned and open-world performance. Self-training yields continued improvements on a suite of diverse datasets (ODinW13;  $x$ -axis), but performance on any given dataset (e.g. LVIS;  $y$ -axis) may saturate (red circles). Fine-tuning on a target dataset improves performance on that dataset, but reduces the open-world generalization ability in proportion to the finetuning duration (light blue squares; numbers indicate finetuning steps). This trade-off can be improved through weight-space ensembling (averaging) of the pretrained and fine-tuned checkpoints [37] (purple diamonds; numbers indicate the mixing coefficient for the fine-tuned weights). The plot shows B/16 models self-trained on N-gram pseudo-annotations and evaluated either directly after self-training or after fine-tuning on  $\mathrm{LVIS}_{\mathrm{base}}$ . Ensembles were created between the longest-self-trained checkpoint and the weights obtained after finetuning that checkpoint for 20k steps. Note that there is significant variability in ODinW13 performance between checkpoints towards the end of self-training.
4.6 Effect of Fine-Tuning on Open-Vocabulary Performance
For contrastively trained image-text models, fine-tuning improves performance on the target distribution but reduces the (originally very high) robustness to distribution shift [30, 28, 37]. We observe the same effect for detection, using ODinW13 AP as a proxy for out-of-distribution performance: Compared to the performance after self-training (red dots in Figure 5), fine-tuning on  $\mathrm{LVIS}_{\mathrm{base}}$  improves performance on the fine-tuned classes (LVIS  $\mathrm{mAP}_{\mathrm{frequent}}$ ), but OOD performance (ODinW13 AP) is simultaneously reduced in proportion to the amount of fine-tuning (light blue line in Figure 5).
A simple approach to improve on this trade-off is to create an ensemble of the model before and after fine-tuning by averaging the model weights [37]. This approach comes at no additional training cost and improves the Pareto-frontier for all ensemble mixing ratios (Figure 5, purple line). We also tried co-training on WebLI and LVIS but found it to perform worse than weight ensembling.
Notably, performance on  $\mathrm{LVIS}_{\mathrm{rare}}$  behaves similarly to  $\mathrm{LVIS}_{\mathrm{frequent}}$  and improves during fine-tuning, even though no  $\mathrm{LVIS}_{\mathrm{rare}}$  classes are seen (Figure 5, right). This may be because  $\mathrm{LVIS}_{\mathrm{rare}}$  classes are semantically and visually close to  $\mathrm{LVIS}_{\mathrm{frequent}}$  classes. For example, seeing many annotations for "bird" may improve performance on rare classes such as "heron", "mallard", or "puffin". LVIS  $\mathrm{mAP}_{\mathrm{rare}}$  therefore only measures a narrow concept of open-vocabulary performance, and does not reveal the fact that fine-tuning significantly reduces generalization to broader distribution shifts. Benchmarks such as ODinW therefore provide significant additional insight.
The main limitation of our method is the amount of compute and data needed for self-training. As we show in Section 4.5, performance improves consistently with training compute and data. This means that further improvements are possible, but also that these will come at increasingly large costs. In fact, cost likely increases faster than resources can realistically be grown in practice. New approaches will therefore be eventually necessary for further improvements.
A second important limitation of our method, similar to other open-vocabulary models [30, 28, 37], is the trade-off between fine-tuned and open-vocabulary performance addressed in Section 4.6. For out-of-distribution queries, predictions of fine-tuned models may be poorly calibrated and may depend on the precise wording of the query. These issues can be mitigated with weight ensembling [37], but more research is needed to fully understand the open-vocabulary robustness of these models.
In the past, open-vocabulary detection performance has been limited by the availability of human-annotated detection training data. Here, we show that self-training can be scaled up to overcome the dependency on human annotations. Our OWL-ST recipe delivers large improvements in detection performance using weak supervision from abundant Web data, similar to what has been seen for image classification and language modelling.
We would like to thank Xiao Wang for help with the WebLI dataset, Xiaohua Zhai and Lucas Beyer for providing the SigLIP model, and Rich Munoz and Alexey Dosovitskiy for insightful comments.
[1] Arandjelović, R., Andonian, A., Mensch, A., Henaff, O.J., Alayrac, J.B., Zisserman, A.: Three ways to improve feature alignment for open vocabulary detection. arXiv preprint arXiv:2303.13518 (2023)
[2] Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 (2022)
[3] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M.J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., Zhang, Q.: JAX: composable transformations of Python+NumPy programs (2018), http://github.com/google/jax
[4] Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A.V., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B.K., Ruiz, C.R., Steiner, A.P., Angelova, A., Zhai, X., Houlsby, N., Soricut, R.: PaLI: A jointly-scaled multilingual language-image model. ICLR (2023)
[5] Dave, A., Dollár, P., Ramanan, D., Kirillov, A., Girshick, R.: Evaluating large-vocabulary object detectors: The devil is in the details. arXiv preprint arXiv:2102.01066 (2021)
[6] Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Riquelme, C., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., van Steenkiste, S., Elsayed, G.F., Mahendran, A., Yu, F., Oliver, A., Huot, F., Bastings, J., Collier, M.P., Gritsenko, A., Birodkar, V., Vasconcelos, C., Tay, Y., Mensink, T., Kolesnikov, A., Pavetic, F., Tran, D., Kipf, T., Lucic, M., Zhai, X., Keysers, D., Harmsen, J., Houlsby, N.: Scaling vision transformers to 22 billion parameters. ICML (2023)
[7] Dehghani, M., Gritsenko, A.A., Arnab, A., Minderer, M., Tay, Y.: SCENIC: A JAX library for computer vision research and beyond. arXiv preprint arXiv:2110.11403 (2021)
[8] Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.Y., Cubuk, E.D., Le, Q.V., Zoph, B.: Simple copy-paste is a strong data augmentation method for instance segmentation. CVPR (2021)
[9] Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 (2021)
[10] Gupta, A., Dollar, P., Girshick, R.: LVIS: A dataset for large vocabulary instance segmentation. CVPR (2019)
[36] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. NeurIPS (2017)
[37] Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Gontijo-Lopes, R., Hajishirzi, H., Farhadi, A., Namkoong, H., Schmidt, L.: Robust fine-tuning of zero-shot models. CVPR (2022)
[38] Yao, L., Han, J., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, H.: Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. arXiv preprint arXiv:2304.04514 (2023)
[39] Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., Xu, H.: DetCLIP: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. NeurIPS (2022)
[40] Yao, Z., Ai, J., Li, B., Zhang, C.: Efficient detr: Improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318 (2021)
[41] Yin, H., Vahdat, A., Alvarez, J., Mallya, A., Kautz, J., Molchanov, P.: Adavit: Adaptive tokens for efficient vision transformer. CVPR (2022)
[42] Zhai, X., Kolesnikov, A., Houlsby, N., Beyer, L.: Scaling vision transformers. CVPR (2022)
[43] Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343 (2023)
[44] Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: LiT: Zero-shot transfer with locked-image text tuning. arXiv preprint arXiv:2111.07991 (2021)
[45] Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li, L.H., Dai, X., Wang, L., Yuan, L., Hwang, J.N., Gao, J.: Glipv2: Unifying localization and vision-language understanding. NeurIPS (2022)
[46] Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al.: RegionCLIP: Region-based language-image pretraining. arXiv preprint arXiv:2112.09106 (2021)
[47] Zhou, X., Girdhar, R., Joulin, A., Krahenbuhl, P., Misra, I.: Detecting twenty-thousand classes using image-level supervision. arXiv preprint arXiv:2201.02605 (2021)
[48] Zoph, B., Ghiasi, G., Lin, T.Y., Cui, Y., Liu, H., Cubuk, E.D., Le, Q.V.: Rethinking pre-training and self-training. NeurIPS (2020)
The Appendix provides a Model Card [27] for OWLv2 as well as additional methodological details, hyperparameters, and results. At the end of the Appendix, we provide qualitative examples of the self-training data and model predictions.
The Appendix is structured as follows:
A.1 Model Card 14
A.2 Human-Curated Label Space 15
A.3 Machine-Generated Label Space 15
A.4 Combined Label Space 16
A.5 Augmentations for Self-Training 16
A.6 Token Dropping 17
A.7 Further Efficiency Improvements 18
A.8 Model Hyperparameters 18
A.9 Additional Results 19
A.9.1 Fixed Average Precision 19
A.9.2 Per-Dataset ODinW Results 19
A.9.3 Fine-Tuning Robustness Trade-Off for OWLv2 L/14 19
A.10 Qualitative Examples 19
A.1 Model Card
Model Summary
Model Architecture	OWL v2 is an open-vocabulary object detector based on OWL-ViT [26]. It consists of an image encoder with a Vision Transformer [17] architecture, a text encoder with a similar Transformer architecture, and heads that predict bounding boxes and label scores from provided images and text queries.
Input(s)	An image and a list of free-text object descriptions (queries).
Output(s)	A list of bounding boxes and a score for each box/query pair.
Usage
Application	The model is intended for open-vocabulary object detection.
Known Caveats	(1) Confidence scores of predictions are not intended to be compared across text queries. While the training loss encourages cross-query calibration for seen queries, scores for unseen queries are not calibrated. Further, the mean Average Precision (mAP) metric does not measure cross-query calibration, so higher mAP does not imply better cross-query calibration. Also see Section 5.(2) Fine-tuning the model creates a trade-off between the performance on fine-tuned texts and unseen texts. See Section 4.6 for details.
System Type
System Description	This is a standalone model.
Upstr. Dependencies	None.
Downstr. Dependencies	None.
Implementation Frameworks
Hardware & Software	Hardware: TPU [13] v2 or v3 (for B- and L-sized models) or v4 (for G-sized models). Software: JAX [3], Flax [11], Scenic [7].
Compute Requirements	Reported in Section 4.5.
Model Characteristics
Model Initialization	The model is initialized from pre-trained language CLIP [30] or SigLIP [43] checkpoints.
Model Status	This is a static model trained on an offline dataset.
Model Stats	The largest OWLv2 model has 2.3B parameters, of which 2B are used for the image encoder and 300M for the text encoder (the heads have a negligible number of parameters). We also trained models with 430M and 150M parameters.
Data Overview
Training dataset	The model is self-trained on bounding boxes predicted by the original OWL-ViT L/14 model [26] on the WebLI dataset [4]. Details on the annotation procedure are provided in Section 3.1.
Evaluation & Fine-tuning Dataset	Open-vocabulary object detection performance is evaluated using the LVIS [10] and ODinW13 [21] datasets.As indicated in Table 1, some models are fine-tuned on the "base" annotations of LVIS, i.e. only annotations for "frequent" and "common" object categories as defined in the official annotations [10]. None of our models have seen any human annotations for LVIS "rare" categories, such that LVIS \( \mathrm{mAP}_{\text{rare}} \) measures zero-shot performance.
Evaluation Results
Evaluation Results	Reported in Table 1.
Model Usage & Limitations
Sensitive Use	The model detects objects matching free-text descriptions. This capability should not be used for unethical use cases such as surveillance.
Known Limitations	Reported in Section 5.
Ethical Considerations	Reported in Section 5.
A.2 Human-Curated Label Space
The human-curated label space was obtained by merging common dataset class lists with the Python code below.
1 # Dataset class names, as available e.g. from TensorFlow Datasets.   
2 # For Visual Genome, we used the 1600 most common label strings.   
3 LVIS_CLASS_NAMES  $=$  [...]   
4 OBJECTS365_CLASS_NAMES  $=$  [..]   
5 OPEN_IMAGE_V4boxesABLE_CLASS_NAMES  $=$  [..]   
6 VISUAL_GENOME_CLASS_NAMES  $=$  [..]   
7 queries  $=$  ( LVIS_CLASS_NAMES + OBJECTS365_CLASS_NAMES + OPEN_IMAGE_V4.BoxABLE_CLASS_NAMES   
12 )   
13   
14 # Remove duplicates: queries  $=$  set([q.lowere() for q in queries])   
16   
17 # Remove plural forms: remove  $=$  set()   
19 for singular in queries: plurals  $=$  [singular  $^+$  's', singular  $^+$  es'] for plural in plurals: if plural in queries: remove.add(plural)   
24   
25 # Same queries for all images: queries  $=$  list(query.difference(remove))
A.3 Machine-Generated Label Space
The machine-generated label space was obtained from the image-associated text, for each image separately, using the Python code below. Figure A3 shows example pseudo-annotations using the N-gram label space.
from typing import Iterator, List   
import nltk   
# Stopwords from nltk.corpus.stopwords词语('english'): STOPWORDS_EN = frozenset({ 'a', 'about', 'above', 'after', 'again', 'against', 'all', 'am', 'an', 'and', 'any', 'are', 'as', 'at', 'be', 'because', 'been', 'before', 'being', 'below', 'between', 'both', 'but', 'by', 'can', 'did', 'do', 'does', 'doing', 'don', 'down', 'during', 'each', 'few', 'for', 'from', 'further', 'had', 'has', 'have', 'having', 'he', 'her', 'here', 'hers', 'herself', 'him', 'himself', 'his', 'how', 'i', 'if', 'in', 'into', 'is', 'it', 'its', 'itself', 'just', 'me', 'more', 'most', 'my', 'myself', 'no', 'nor', 'not', 'now', 'of', 'off', 'on', 'once', 'only', 'or', 'other', 'our', 'ours', 'ourselves', 'out', 'over', 'own', 's', 'same', 'she', 'should', 'so', 'some', 'such', 't', 'than', 'that', 'their', 'theirs', 'them' , themselves, then, there, these, they, this, those, through, to, too, under, until, up, very, was, we, were, what, when, where, which, while, who, whom, why, will, with, you, your, yours, yourself, yourselves   
}   
These words were found by manually going through the most common 1000 words # in a sample of alt-texts and selecting generic words without specific meaning: COMMON.GenericWORDS = frozenset({ alibaba'aliexpress'amazon'available'background'blog'buy'