SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matthias Minderer
Alexey Gritsenko
Neil Houlsby
{mjlm, agritsenko, neilhoulsby}@google.com
Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales ( $\approx$ 10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With a ViT-L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from  $31.2\%$  to  $44.6\%$  (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling. Code and checkpoints are available on GitHub.<sup>1</sup>
Object detection is a core computer vision task with many real-world applications. Consequently, there is great interest in improving detection models, especially in the open-vocabulary domain. For image-level tasks, large improvements have been achieved through contrastive pretraining of vision-language models, which is massively scalable because it can use naturally abundant weak supervision in the form of image-text pairs from the Web [30, 12, 29]. Since no such natural supervision data exists for localization tasks, open-vocabulary detection models typically build on pretrained image-level encoders [9, 19, 26, 46, 22, 1, 39, 47]. However, due to the scarcity of detection data and the fragility of pretrained representations, detection-training stages of these models have typically had to be relatively brief, which limits final detection performance and scaling potential.
The scarcity of detection data can be addressed with self-training. In self-training, an existing detector is used to predict bounding boxes on unlabeled images to generate data for training better detectors [31, 48, 35]. By combining open-vocabulary detectors with Web image-text data, such pseudo-labeling can produce practically unlimited amounts of open-vocabulary detection training data that leverages the image-associated text for semantic supervision. While several works have applied various forms of self-training to open-vocabulary object detection [46, 1, 47, 39, 38], they have done so at relatively small scales, comparable to the size of human-annotated detection datasets and much smaller than the datasets used for image-level training.
<https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit>
Figure 1: Overview of our method. Left: Our method has three steps: (1) Generate pseudo-box annotations on WebLI with OWL-ViT L/14, queried with caption N-grams. (2) Train new models on pseudo-annotations. (3) Optionally, fine-tune on human annotations. Right: Zero-shot detection performance on  $\mathrm{LVIS}_{\mathrm{rare}}$  after fine-tuning on  $\mathrm{LVIS}_{\mathrm{base}}$ . Neither the annotator nor our models have seen any human-generated box annotations for  $\mathrm{LVIS}_{\mathrm{rare}}$  classes. Our self-training approach improves over other methods even at moderate amounts of training (e.g., the OWL-L/14 model we use as annotator; black  $\times$ ), and continues to improve as training is scaled up. Horizontal black lines indicate previous state-of-the-art open-vocabulary detectors which did not see  $\mathrm{LVIS}_{\mathrm{rare}}$  classes during training.
To scale detection self-training further, we take guidance from image-level methods, where the principle has been to leverage weak supervision in the largest possible amount [30, 12, 29, 42]. We identify three key ingredients for optimizing the use of weak supervision for detection: choice of label space, filtering of pseudo-annotations, and training efficiency. Prior methods have typically used human-curated label spaces or complex concept mining [47, 39, 38, 46] and strict filtering, keeping just the single largest [47] or highest-scoring [1] pseudo-box for each image. In contrast, we argue that we should "let the data do the work" and therefore apply little processing and filtering. We propose to simply use all possible N-grams of the image-associated text as detection prompts for that image, and apply only weak confidence filtering to the resulting pseudo-labels.
We apply this self-training recipe to the OWL-ViT detection architecture [26] and call it OWL-ST. To increase the number of examples seen for a given amount compute, we also introduce OWLv2, an optimized architecture with improved training efficiency. Combining the OWL-ST recipe with the OWLv2 architecture surpasses prior state-of-the-art methods already at moderate amounts of self-training, comparable to training amounts of previous methods (Figure 1). Scaling self-training to billions of examples yields further large improvements. For example, our ViT-L/14-based model, trained on 2.3B image-text pairs and fine-tuned on LVISbase, achieves  $44.6\%$  zero-shot LVIS  $\mathrm{mAP}_{\mathrm{rare}}$  which is a  $36\%$  relative improvement over the prior state of the art ( $32.8\%$ $\mathrm{mAP}_{\mathrm{rare}}$  for F-VLM R50x64 [19]). Our largest model, ViT-G/14, reaches  $47.2\%$ $\mathrm{mAP}_{\mathrm{rare}}$ .
We also evaluate our models on a suite of "in the wild" datasets [21] and study the trade-off between fine-tuned and open-vocabulary performance. We find that strong in- and out-of-distribution performance is possible with weight ensembling [37]. Finally, our analysis of the scaling behavior of OWL-ST suggests that self-training has further potential for leveraging abundantly available weak supervision for open-vocabulary object detection.
2.1 Scaling Vision Models
Vision models have recently seen large advances in model and training scale, leading to improved performance on many image-level tasks. On the architecture side, Vision Transformers have been shown to scale more efficiently than prior architectures [17]. Task performance improves predictably as training data and compute are increased [42], with recent work showing continued improvements for models with up to 22 billion parameters [6]. We apply these findings to object detection.
On the data side, contrastive pretraining of vision-language models (VLMs) [30] has unlocked the use of abundantly available image-text pairs from the Web as weak supervision, with improved results if more data is used [12, 28]. VLMs, which embed images and text into a shared space, also enable open-vocabulary applications where prior models were limited to fixed label spaces. Here, we use pretrained CLIP [30] and SigLIP [43] encoders as backbones for our detector.
2.2 Open-Vocabulary Object Detection
Much recent work aims to transfer the open-vocabulary capabilities of VLMs to localization tasks such as object detection. A first wave of VLM-based object detection methods either distilled VLM-predictions for cropped image regions (e.g. ViLD [9]), or added detection heads directly to frozen (F-VLM [19]) or fine-tuned (OWL-VIT [26]) VLM encoders. A challenge identified by these works is to protect the VLM from forgetting its open-vocabulary knowledge while training the detection heads on the relatively little available detection data.
2.3 Scaling Open-Vocabulary Detection with Weak Supervision
Given that earlier methods identified detection data as a limiting factor in open-vocabulary detection performance, more recent works focus on using weak supervision directly for detection training, rather than just during VLM pretraining. There are two main approaches:
Some methods use self-training, in which an existing detector is used to predict pseudo-boxes for images where image-level labels or captions, but no human box annotations, are available. Better detectors can then be trained on the pseudo-annotations. For example, RegionCLIP [46] generates pseudo-boxes using nouns parsed from image captions and uses those boxes for localization pretraining. Detric [47] predicts class-agnostic pseudo-boxes on images for which classification labels are available and associates the largest predicted box with the image label. Similar to our approach, 3Ways [1] uses an existing open-vocabulary detector to predict pseudo-boxes on captioned images, but uses the whole caption as a prompt, instead of dividing it into multiple prompts as we do.
Other methods propose grounding losses that directly train a detector on weak supervision such as image-level labels or captions. These methods pretrain models to align class-agnostic pseudo-boxes with words from image-associated text and rely on human-generated detection data for fine-tuning. Major examples of this approach are GLIPv1/v2 and [22, 45] and DetCLIPv1/v2 [39, 38].
In principle, these approaches unlock Web-scale training for detection, but prior methods rarely go much beyond 10M examples and instead focus on the model architecture and training loss. Here, we keep architecture and loss simple, and focus on scaling up the training data, since this was successful for image-level models. A similar approach was recently applied with good results to class-agnostic segmentation in the Segment Anything work [16]. Together with our results on text-conditioned localization, this suggests that scaling up self-training is a powerful and general method for improving performance on fine-grained vision tasks.
We propose a simple self-training approach with three steps: (1) Use an existing open-vocabulary detector to predict bounding boxes for a large Web image-text dataset. (2) Self-train a new detector on the pseudo-annotations. (3) Optionally, fine-tune the self-trained model briefly on human-annotated detection data (Figure 1, left). Our goal is to optimize the key components of this approach—label space, annotation filtering, and training efficiency—such that it provides strong and scalable open-vocabulary performance with few human annotations.
3.1 Generating Web-Scale Open-Vocabulary Object Annotations
We use the WebLI dataset [4] as the source of weak supervision for self-training. WebLI is a large dataset of images and texts available on the public Web. The dataset consists of approximately 10B images and associated alt-text strings, which can be thought of as noisy image captions. For images whose alt-text is not in English, we use an automatically generated English translation [4].
We use OWL-ViT CLIP-L/14 [26] to annotate all 10B WebLI images with bounding box pseudo-annotations. OWL-ViT is an open-vocabulary object detector. Given an image, the model first detects objects in the image in a class-agnostic way. Then, given a list of free-text queries, the model produces scores indicating the likelihood that each detected object is associated with each text query.
A crucial design choice for open-vocabulary pseudo-labeling is the annotation label space. Methods in the literature vary widely but typically fall somewhere between two extremes: (1) use a fixed, human-curated label space for all images (e.g. [47]), or (2) machine-generate per-image queries from image-associated text (e.g. [1]). We implement both and compare their performance in Section 4.3.
Human-curated label space. We performed one pseudo-annotation run by combining the label sets from the LVIS [10], Objects365 [33], OpenImagesV4 [20], and Visual Genome [18] datasets and removing duplicates and plural forms. In total, this label space contains 2520 common object categories, e.g. "phone", "goatee", "teakettle", "park", "suit (clothing)". See Appendix A.2 for code to generate the full list. Models trained on this label space may not be considered fully open-vocabulary for evaluation datasets whose classes were included in the pseudo-annotation label space (e.g. LVIS), since the evaluation vocabulary is known at training time in this case. However,  $\mathrm{LVIS}_{\mathrm{rare}}$  classes are still unseen for all of our models, in the sense that neither the annotator nor the self-trained models have ever seen human box annotations for  $\mathrm{LVIS}_{\mathrm{rare}}$  classes.
Machine-generated label space. In a second pseudo-annotation run, we automatically generated queries from the image-associated text. Prior work using image captions as weak supervision for detection often used grammatical parsing to extract noun phrases or concepts [46, 39, 38]. These approaches may add biases that reduce the diversity of extracted queries. To keep such biases to a minimum, we use no grammatical parsing and simply extract all word N-grams up to length 10 from the text associated with a given image and use them as queries for that image. We apply minimal filtering, only removing generic terms like image or png, and queries consisting entirely of stop-words (details in Appendix A.3). Note that, since OWL-ViT uses late image-text fusion, the quality of box localization (as opposed to classification) is not affected by the chosen label space.
Regardless of label space, we ensemble predictions over seven prompt templates such as "a photo of a {}" as described in [26]. For each predicted box, we keep the query with the highest score as its pseudo-label. For each image, we keep all boxes above a score threshold. We study the choice of threshold in Section 4.4. The pseudo-annotations are used as hard labels for self-training.
3.2 Self-training at Scale
We now describe how we use the pseudo-annotations to train better detectors. We use a variant of the OWL-ViT architecture [26] as described below. The image and text encoders are initialized from contrastively trained image-text models (CLIP, unless noted otherwise); the detection heads are randomly initialized. All models are first trained exclusively on pseudo-annotations ("self-training"). In an optional separate step, models are fine-tuned briefly on human-annotated detection data.
Self-training proceeds similarly to detection training in [26]. In particular, we use the same losses and also augment queries with "pseudo-negatives" that are randomly sampled from the queries of other images, similar to batch negatives in [1]. Due to the size of our dataset, in contrast to [26], we use no random prompt templates and fewer image augmentations (details in Appendix A.5).
Prior work on image-level tasks shows that pretraining improves performance on downstream tasks well beyond 1 billion examples seen [44, 12, 28, 42], across model sizes. We hypothesize that similar scaling applies to detection self-training. We therefore optimize training efficiency to maximize the number of images seen for a given amount of training compute as follows.
Token dropping. Vision Transformers represent images as an unordered sequence of tokens. Tokens can therefore be reorganized or dropped without changing the model parameters. Various forms of token dropping or pooling have been proposed to improve efficiency [24, 32, 41, 25, 2]. Here, we drop tokens simply based on the pixel variance of the corresponding image patch. Both natural and Web images contain low-variance areas devoid of useful information, e.g. sky, single-color backgrounds, or padding. We find that the lower half of image patches by mean pixel variance can be dropped without loss in detection performance (Appendix A.6). We therefore drop  $50\%$  of patches in all of our experiments during training. No patches are dropped during inference.
Table 1: Open-vocabulary detection performance on LVIS and ODinW. Rows for our models are shown in blue. None of our models have seen any human box annotations for  $\mathrm{LVIS}_{\mathrm{rare}}$  classes at any stage of training, so  $\mathrm{LVIS~AP}_{\mathrm{rare}}^{\mathrm{val}}$  (rightmost column) measures zero-shot performance. Numbers in green or red indicate the difference to the prior state of the art, i.e. F-VLM R50x64 in the open-vocabulary (top) part of the table and DetCLIPv2 Swin-L in the curated-vocabulary (bottom) part. Gray O+VG indicates that O365+VG were used indirectly (for training the annotator). Gray ODinW numbers indicate that these models were trained on OpenImages data, which overlaps with ODinW.  $\mathrm{AP}^{\mathrm{mini}}$  refers to the LVIS "minimal" split introduced by MDETR [14].
Method	Backbone	Self-training data	Self-training vocabulary	Human box annotations	ODinW 13	LVIS \(AP_{all}^{mini}\)	LVIS \(AP_{rare}^{mini}\)	LVIS \(AP_{all}^{val}\)	LVIS \(AP_{rare}^{val}\)
Open vocabulary (evaluation vocabulary is not available at training time):
1	RegionCLIP [46]	R50x4	CC3M	6k concepts	LVISbase	-	-	-	32.3	22.0
2	OWL [26]	CLIP B/16	-	-	O365+VG	-	-	-	27.2	20.6
3	OWL [26]	CLIP L/14	-	-	O365+VG	48.4	-	-	34.6	31.2
4	GLIPv2 [45]	Swin-T	Cap4M	tokens	O365+GoldG	48.5	29.0	-	-	-
5	GLIPv2 [45]	Swin-B	CC15M	tokens	FiveODs+GoldG	54.2	48.5	-	-	-
6	GLIPv2 [45]	Swin-H	CC15M	tokens	FiveODs+GoldG	55.5	50.1	-	-	-
7	F-VLM [19]	R50x4	-	-	LVISbase	-	-	-	28.5	26.3
8	F-VLM [19]	R50x64	-	-	LVISbase	-	-	-	34.9	32.8
9	3Ways [1]	NFNet-F0	TODO	captions	LVISbase	-	-	-	35.7	25.6
10	3Ways [1]	NFNet-F6	TODO	captions	LVISbase	-	-	-	44.6	30.1
11	OWL-ST	CLIP B/16	WebLI	N-grams	O+VG	48.8	31.8	35.4	27.0	29.6-3.2
12	OWL-ST	CLIP L/14	WebLI	N-grams	O+VG	53.0	38.1	39.0	33.5	34.9+2.1
13	OWL-ST	SigLIP G/14	WebLI	N-grams	O+VG	49.9	37.8	40.9	33.7	37.5+4.7
14	OWL-ST+FT	CLIP B/16	WebLI	N-grams	O+VG, LVISbase	48.6	47.2	37.8	41.8	36.2+3.4
15	OWL-ST+FT	CLIP L/14	WebLI	N-grams	O+VG, LVISbase	50.1	54.1	46.1	49.4	44.6+11.8
16	OWL-ST+FT	SigLIP G/14	WebLI	N-grams	O+VG, LVISbase	50.1	51.3	50.9	47.0	47.2+14.4
Human-curated vocabulary (evaluation vocabulary may be accessed at training time):
17	Detic [47]	R50	IN-21k	LVIS classes	LVISbase	-	-	-	32.4	24.6
18	DetCLIPv2 [38]	Swin-T	CC15M	Nouns+curated	O365+GoldG	-	40.4	36.0	32.8	31.0
19	DetCLIPv2 [38]	Swin-L	CC15M	Nouns+curated	O365+GoldG	-	44.7	43.1	36.6	33.3
20	OWL-ST+FT	CLIP B/16	WebLI	N-grm+curated	O+VG, LVISbase	48.9	51.1	41.9	45.6	40.5+7.2
21	OWL-ST+FT	CLIP L/14	WebLI	N-grm+curated	O+VG, LVISbase	48.7	55.8	50.0	50.4	45.9+12.6
Instance selection. OWL-ViT is an encoder-only architecture and predicts one bounding box per encoder token. This is inefficient, since there are typically many more encoder tokens than objects (e.g. 5184 tokens for resolution  $1008 \times 1008$  and patch size  $14 \times 14$ ). Most output tokens therefore do not represent objects. We introduce an objectness head which predicts the likelihood that an output token actually represents an object, and compute boxes, class scores, and losses only for the top  $k$  tokens by objectness, similar to Efficient DETR [40]. The objectness head receives an encoder token as input and computes a scalar objectness score. The objectness score predicts the future classification score of a token and is supervised by the actual classification score of those tokens that end up being selected and passed on to the classification head. We select approximately  $10\%$  of instances by top objectness during training in all of our experiments. During inference, all instances are used.
Mosaics. During self-training, we combine raw images into grids of up to  $6 \times 6$  to produce a single training example (i.e. a more extreme version of the mosaics in [26]). This has two main motivations: (1) Using mosaics increases the number of raw images seen for a given fixed model input resolution. An alternative is to train using variable image sizes [38], but this would require resizing image position embeddings for each input size. (2) The average resolution and complexity of Web images is lower than images in detection benchmarks and applications. Mosaics reduce the average object size and improve small-object performance, similar to large scale-jittering [8], but with less padding. For all self-training experiments, we use  $1 \times 1$ ,  $2 \times 2$ ,  $3 \times 3$ ,  $4 \times 4$ , and  $6 \times 6$  grids in equal proportions, resulting in an average of 13.2 raw component images per training example.
To further improve training efficiency, we also adopt previously proposed practices for large-scale Transformer training [42] (details in Appendix A.7). Together, our improvements reduce training FLOPS by approximately  $50\%$  compared to the original OWL-ViT [26] and increase training throughput by  $2\times$  (e.g. for L/14 at  $840\times 840$  resolution measured on TPUv3: GFLOPs/example 11'945.4 vs. 5357.9; examples/s/core 1.0 vs. 2.2). We refer to the improved model as OWLv2.
At inference, no token dropping or instance selection is performed. Inference is therefore identical to the original OWL-ViT, i.e. each image encoder token is decoded into a bounding box and a list of per-query classification scores.
Figure 2: Comparison of pseudo-label spaces. Self-training on a human-curated list of classes yields good downstream performance on these classes, but generalizes poorly to unseen classes and datasets. Open-vocabulary generalization can be improved by obtaining weak but diverse supervision from image-associated text. WebLI image-text data was pseudo-annotated using OWL-ViT CLIP-L/14 with one of three label spaces: Curated vocabulary (the union of label spaces from LVIS, Objects365, OpenImagesv4, and Visual Genome),  $N$ -grams (lightly filtered N-grams from the text associated with each image), or a combination of both ( $N$ -grams + curated). OWLv2-B/16 models were then self-trained on the pseudo-annotations and fine-tuned on  $\mathrm{LVIS}_{\mathrm{base}}$ . Each point represents a separate fine-tuning run. "Examples seen" refers to the number of images after creating mosaics; the total number of raw images seen is  $13.2 \times$  that number (Section 3.2).
3.3 Fine-tuning
Self-training on pseudo-annotations alone already yields strong performance (Section 4.2). However, fine-tuning briefly on human annotations can provide significant further benefits. For fine-tuning, we start with the learning rate and optimizer state of the self-trained checkpoint and then continue training on the target dataset while linearly cooling down the learning rate to zero. Fine-tuning of open-vocabulary models involves a trade-off between improving the performance on the fine-tuned classes and losing open-vocabulary performance [30, 29, 37]. We study this trade-off in Section 4.6.
Models. We use the publicly available OWL-ViT CLIP L/14 model to generate detection pseudo-annotations for the WebLI dataset (10 billion image-text pairs [4]). For all self-training experiments, we use OWL-ViT models modified as described in Section 3.2. Backbones are initialized with the publicly available CLIP [30] checkpoints (B/16 and L/14) or a SigLIP [43] checkpoint (G/14).
Training. Models are first self-trained on the pseudo-annotations for varying durations as indicated. If indicated, after self-training, models are fine-tuned on  $\mathrm{LVIS}_{\mathrm{base}}$ , i.e. the LVIS dataset [10] with all annotations for "rare" categories removed. Therefore, neither the annotator nor any of our models have seen human-generated annotations for  $\mathrm{LVIS}_{\mathrm{rare}}$  classes. Fine-tuning uses mosaics up to  $3 \times 3$  and is always done until the model has seen 256'000 mosaics (1.1M individual images, roughly equivalent to 100 LVIS epochs). The image size is  $960 \times 960$  for /16 models and  $1008 \times 1008$  for /14 models. See Appendix A.8 for a complete list of hyperparameters.