Scaling Open-Vocabulary Object Detection
Authors: Matthias Minderer
Alexey A. Gritsenko
Neil Houlsby
Venue: NeurIPS 2023 spotlight (2023)
Keywords: object detection
open-vocabulary object detection
vision transformers
vision-language models
scaling
self-training
TL;DR: We scale self-training for open-vocabulary object detection to billions of examples, leading to massive improvements in detection performance.

Abstract: Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling. Code and checkpoints are available on GitHub.

Scaling Open- Object Detection
Matthias Minderer
Alexey Gritsenko
Neil Houlsby
Google DeepMind
[mjlm, Lm, agritsenko, neilhoulsbyylggoggee.ccm
Abstract
Open-vocabulary object detection has benefited greatly from pretrained vision- language models, but still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image- level pretraining. Here, scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo- annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-ate art open- open-vocabulary detectors already at comparable training scales (=10M examples). However, with OWL-ST, we can scale to over IB examples, yielding further large improvement: With a ViT- 1/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open- world localization, similar to what has been seen for image classification and language modelling. Code and checkpoints are available on GitHub.'
Introduction
Object detection is a core computer vision task with many real-world applications. Consequently, there is great interest in improving detection models, especially in the open- vocabulary domain. For image-level tasks, large improvements have been achieved through contrastive pretraining of vision- naturally abundant weak supervision in the form of image-text pairs from the Web [30, 12, 29]. Since no such natural supervision data exists for localization tasks, open-voca detection models typically build pretrained image- level encoders [9, 19, 26, 46, 22, 1, 39, 47]. However, due to the scarcity of detection data and the fragility of pretrained representations, detection-training stages of these models have typically had to be relatively brief, which limits final detection performance and scaling potential.
language models, which is massively scalable because
The scarcity of detection data can be addressed with self-training. In self-training, existing detector is used to predict bounding boxes on unlabeled images to generate data for training better detectors [31, 48, 35]. By combining open- poccbbocay detectors with Web image-text data, such pseudo-labeling can produce practically unlimited amounts of open-vocabulary detection training data that leverages the image-associated text for semantic supervision. While several works have applied various forms of self-training to open-vocabulary object detection [46, 1, 47, 39,38) 38), they have done at relatively small scales, comparable the size of numan-annotated detection datasets and much smaller than the datasets used for image-level training.
vit
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1. Annotation
2. Self-training
3. Fine-tuning (sptional) LMIS how
Unseen classes
Caption:
Pseudo annotated WibLi
Stnesa
Negramet
Model
OWL-ST+FT G/14
LV15 AP..
40
OWL-ST+FT L/14
Open phe seunn
35
OWL-ST+FF B/16
OWI.1/14 (annotator)
Peggt DWM
OW +7
ON
30
VILD
OWLVI
OWL-VTT
10'
10P
10
Total examples seen (including repetitions)
Figure 1: Overview of our method. Left: Our method has three steps: (1) Generate pseudo-bo box annotations on WebLI with OWL-ViT L/14, queried with caption N N-grams. (2) Train new models on psoudo-annotations. (3) Optionally, fine-tune on human annotations, Right: Zero-shot detection performance on LVIS after fine-tuning on LVIS Neither the annotator nor our models have seen any human-genn generated box annotations for LVIS,are classes. Our self self-training approach improves over other methods even moderate amounts of training (e.g. the OWL-L/14 model we use as annotator; black x), and continues to improve as training is scaled up. Horizontal black lines indicate previous state-of-the-art open-vocabulary detectors which did not see LVTS classes during training.
To scale detection self-training further, we take guidance from image-level methods, where the principle has been to leverage weak supervision in the largest possible amount [30, 12, 29, 42]. We identify three key ingredients for optimizing the use of weak supervision for detection: choice of label space, filtering of pseudo-annotations, and training efficiency. Prior methods have typically used human-curated label spaces or complex concept mining [47, 39, 38, 46] and strict filtering, keeping just the single largest [47] or highest-scoring [1] pseudo-box for each image. In contrast, we argue that we should "let the data do the work" and therefore apply little processing and filtering. We propose to simply use all possible N-grams of the image- associated text as detection prompts for that image, and apply only weak confidence filtering to the resulting pseudo-labels.
We apply this self- training recipe to the OIL-VIT detection architecture [26] and call it OWL-ST. To increase the number of examples seen for given amount compute, we also introduce OWLx2, an optimized architecture with improved training efficiency. Combining the OWL-ST recipe with the OWLv2 architecture surpasses prior state-of-the-an methods already at moderate amounts of self-training, comparable to training amounts of previous methods (Figure 1). Scaling self-training to billions of examples yields further large improvements. For example, our ViT-L/14-based model, trained on 2.3B image-text pairs and fine-tuned LVIS achieves 44.6% zero-shot LVIS mAP rare. which is 36% relative improvement over the prior state of the art (32.8% mAP rare for F-VLM R50x64 [19]). Our largest model, ViT-G/14, reaches 47.2% mAP
We also evaluate our models on a suite of "in the wild" datasets [21] and study the trade-off between fine-tuned and vocabuocay performance. We find that strong in- and out-of-distribution per- formance is possible with weight ensembling [37]. Finally, our analysis of the scaling behavior of OWL-ST suggests that self-training has further potential for leveraging abundantly available weak supervision for open-vocabulary object detection.
2
Related Work
2.1 Scaling Vision Models
Vision models have recently seen large advances in model and training scale, leading to improved performance on many image-level tasks. On the architecture side, Vision Transformers have been shown to scale more efficiently than prior architectures [17]. Task performance improves predictably as training data and compute are increased [42], with recent work showing continued improvements for models with up to 22 billion parameters [6]. We apply these findings object detection.

---

On the data side, contrastive pretraining of Vision-language models (VLMs) [30] has unlocked the use of abundantly available image-text pairs from the Web as weak supervision, with improved results more data is used [12, 28]. VLMs, which embed images and text into shared space, also enable open-vocabulary applications where prior models were limited fixed label spaces. Here, we use pretrained CLIP [30] and SigLIP [43] encoders as backbones for our detector.
2.2 Open-Vocabulle Object Detection
Much recent work aims to transfer the open- vren-vocabulary capabilities of VLMs to localization tasks such as object detection. A first wave of VLM- object detection methods either distilled VLM- predictions for cropped image regions (c.g. ViLD [9]), or added detection heads directly to frozen (F-VLM [19]) or fine-tuned (OWL-VIT) [26]) VLM encoders. A challenge identified by these works is to protect the VLM from forgetting its open-vocabulans knowledge while training the detection heads on the relatively little available detection data.
2.3 Scaling Bgen-Vocabulary Detection with Weak Supervision
Given that earlier methods identified detection data as limiting factor in open-vocabulary detection performance, more recent works focus on using weak supervision directly for detection training. rather than just during VLM pretraining. There are two main approaches:
Some methods use self-training, in which existing detector is used to predict pseudo-boxes for images where image-level labels or captions, but no human box annotations, are available. Better detectors can then be trained on the bseudo-annotatio
For example, RegionCLIP [46] generates pseudo-boxes using nouns parsed from image captions and uses those boxes for localization pre- training. Detic [47] predicts class-agnostic pseudo-boxes on images for which classification labels are available and associates the largest predicted box with the image label. Similar to our approach, Wayss [1] uses an existing open- vocabulary detector to predict pseudo-boxes on captioned images, but uses the whole caption as prompt, instead of dividing it into multiple prompts as we do.
Other methods propose grounding losses that directly train detector weak supervision such as image-level labels or captions. These methods pretrain models to align class-agnostic pseudo- boxes with words from Image-associag Major examples of this approach are GLIPv1/v2 and [22, 45] and DetCLIPv1/v2 [39, 38].
text and rely human- gennaanee detection data for fine-tuning.
In principle, these approaches unlock Web-scale training for detection, but prior methods rarely go much beyond 10M examples and instead focus on the model architecture and training loss. Here, we keep architecture and loss simple, and focus on scaling up the training data, since this was successful for image-level models. A similar approach was recently applicd with good results to class-agnostic segmentation in the Segment Anything work [16]. Together with our results text-conditioned localization, this suggests that scaling up self-training powerful and general method for improving performance on fine-gre tne-grained vision tasks.
3 Method
We propose simple self-training approach with three steps: (1) Use an existing open-vocabulary
detector predict bounding boxes for large Web image-text dataset. (2) Self-train a new detector on the pseudo-annotations. (3) Optionally, fine-tune the self-trained model briefly on human-annotated detection data (Figure left). Our goal is to optimize the key components of this approach- label space, annotation filtering, and training efficiency-suce that it provides strong and scalable open-vocabulary performance with few human annotations.
3.1 Generating Web-Scale
Object Annotations
We use the WebLI dataset [4] as the source of weak supervision for self-training. WebLI is large dataset of images and texts available on the public Web. The dataset consists of approximately 10B images and associated alt-text strings, which can be thought of as noisy image captions. For images whose alt-text is not in English, we an automatically generated English translation [4].
3

---

We use OIL-ViT CLIP-L/14 [26] to annotate all 10B WebLI images with bounding box pseudo- annotations. OWL-VVT is an ppen-vocabulary object detector. Given an image, the model first detects objects in the image in class-agnostic way. Then, given a list of free-text queries, the model produces scores indicating the likelihood that each detected object is associated with each text query.
A crucial design choice for open-vocabulary pseudo-labeling is the annotation label space. Methods in the literature vary widely but typically fall somewhere between two extremes: (1) use fixed, human-curated label space for all images (e.g. [47]), or (2) machine-generate per-image queries from image-associated text (e.g. [T]). We implement both and compare their performance Section 4.3.
Human-curated label space. We performed one pseudo pseudo-annotaliee annotation run by combining the label sets from the LVIS [10], Objects365 [33], OpenImages V4 [20], and Visual Genome [18] datasets and removing duplicates and plural forms. In total, this label space contains 2520 common object cate- gories, e.g. "phone", "goatee", "teakettle", "park", "suit (clothing) ". See Appendix A.2 for code to generate the full list. Models trained on this label space may not be considered fully open-vocabulary for evaluation datasets whose classes were included in the pseudo-annotation label space (e.g. LVIS), since the evaluation vocabulary is known at training time in this case. However, LVIS classes are still unseen for all of our models, in the sense that neither the annotator nor the self-trained models have ever seen human box annotations for LVIS classes
Machine-generater label space. In a second pseudo annotation run, automatically generated queries from the image- aasee-assocated text. Prior work using image captions as weak supervision for detection often used grammatical parsing to extract noun phrases concepts [46, 39, 38]. These approaches may add biases that reduce the diversity of extracted queries. To keep such biases to minimum, we no grammatical parsing and simply extract all word N-grams up to length 10 from the text associated with a given image and use them as queries for that image. We apply minimal filtering, only removing generic terms like image or png, and queries consisting entirely of stop-words (details in Appendix A.3). Note that, since OWL-ViT uses late image-text fusion, the quality of box localization (as opposed classification)
affected by the chosen label space.
Regardless of label space, we ensemble predictions over seven prompt templates such as "a photo of O)" described in [26]. For each predicted box, we keep the query with the highest score as its labud.-label. For each image, we keep all boxes above a score threshold. We study the choice of used hard labels for self-training.
threshold in Section 4.4. The pseudo-annotateo
3.2 Self-training at Scale
We now describe how we use the pseudo-annotations train better detectors. We use variant of the OWL-ViT architecture [26] as described below. The image and text encoders are initialized from contrastively trained image-text models (CLIP, unless noted otherwise); the detection heads randomly initialized. All models are first trained exclusively on pseudo- annotaaonnss ("self-training In an optional separate step, models are fine-tuned briefly on human-annotated detection data.
Self-training proceeds similarly to detection training in [26]. In particular, we use the same losses and also augment queries with sneeudo-negative other images, similar to batch negatives in [1]. Due to the size of our dataset, in contrast to [26], we use no random prompt templates and fewer image augmentations (details in Appendix A.5).
that randomly sampled from the queries of
Prior work on image-level tasks shows that pretraining improves performance on downstream tasks well beyond 1 billion examples seen [44, 12, 28, 42), across model sizes. We hypothesize that similar scaling applics to detection self-training. We therefore optimize training efficiency to maximize the number of images seen for given amount of training compute follows.
Token dropping. Vision Transformers represent images an unordered sequence of tokens. Tokens can therefore be reorganized or dropped without changing the model parameters. Various forms of token dropping or pooling have been proposed to improve efficiency [24, 32, 41, 25, 2]. Here, we drop tokens simply based on the pixel variance of the corresponding image patch. Both natural and Web images contain low-variance areas devoid of useful information, e.g. sky, single-color backgrounds, or padding. We find that the lower half of image patches by mean pixel variance can be dropped without loss in detection performance (Appendix A.6). We therefore drop 50% of patches in all of our experiments during training, No patches are dropped during inference.

---

Table 1: Open-vocabulary detection performance on LVIS and WDinW. Rows for our models are shown in blue. None of our models have seen any human box annotations for LVIS classes at any stage of training, in green or red indicate the difference to the prior state of the art, i... F-VLM R50x64 in the open- vocabulary (top) part of the table and DetCLIPv2 Swin-L in the curated- vocabulary (bottom) part. Gray O+VG indicates that O365+VG were used indirectly (for training the annotator). Gray WDin numbers indicate that these models were trained on OpenImages data, which overlaps with ODinW. Ap refers to the LVIS "minival" split introduced by MDETR [14].
LVIS AP (rightmost column) measures zero-shot performance. Numbers
Self-training
Self-training vocabulary
Human box annotations
ODiaW
LVIS AP
LVIS APaini
LVIS AP
LVIS AP
Method
Backhone
data
Open vocahulary (evaluation vocabulary available at training timej:
RegionCLIP [46]
R50x4
CC3M
6k concepts
LVIS
32.3
22.0
2
OWL [26]
CLIP B/16
O365-VG
27.2
20.6
OWL [26]
CLIP 1/14
V365+VG
34.6
4
GLIPv2 [45]
Swin-T
Cap4M
tokens
G3d5+GoldG
48.5
29.0
GLIPP2 [45]
Swin-B B
CC15M
tokens
OiveODs+GioddG
54.2
48.5
GLIPv2 [45]
Swin-H
CCISM
tokens
FiveODs+GoldG
55.5
50.1
F-VLM [19]
R50x4
LVIS
28.5
26.3
8 F-LLM [19]
R50x64
L
34,9
32.8
3Ways
NENet-FO
TODO
captions
LVIS
35.7
25.6
10 3Ways [1]
NHNet-H6
TODO
captions
LV
44,6
30.1
11 ST
CLIP B/16
WebLI
N-grams
O&VG
48.8
31.8
35.4
27.0
29.6 -3.2
OWL-ST
CLIP L/14
WebLI
N-grams
O+VG
53.0
38.1
39.0
33.5
34.9 +2.1
13 OWL-ST
SigLIP G/14
WebLI
N graam
V+VG
49.9
37.8
40.9
33.7
37.5 +4.7
STFF
CLIP B/16
WebLI
N-grams
V+VG, LVIS
LVVSS.VI........
48.6
47.2
37.8
41.8
36.2 +3.4
OWL-ST+FT
CLIP L/14
WebLI
N-grams
O+VG, LVTS
50.1
54.1
46.1
49.4
44.6
16 OWL-ST+FT
IPgLIP G/14
WebLI
N-grams
O+VG, LV1S
50.1
51.3
50.9
47.0
47.2 +14.4
Human-curated vocabulary (evaluation vocabulary may be acceased at training time):
Detic [47]
R50
IN-21K
LVIS classes
LVVS
32.4
24.6
18 DeICLIP22 [38]
Swin-T
CC15M
Nouns-cuuratd
0365+GoldG
40.4
36.0
32.8
31.0
19 DeICLIPv2 [38]
Swin-L
CC15M
Nouns-curated
0365+GoldG
44.7
43.1
36.6
33.3
]
OWL-ST+FT
WebLI
N-grm+curated
OFVG, L
51.1
41.9
45.6
40.5
47.2
21 OWL-ST+FT
L/I4
WebLI
N- N-grm+curated
O+VG, LVTS
48.7
55.8
50.0
50.4
45.9 +12.6
Instance selection. OiL-VIT is an encoder-only architecture and predicts one bounding box per encoder token. This is inefficient, since there are typically many more encoder tokens than objects (1.4.5184 5184 tokens for resolution 1008 1008 and patch size 14 14). Most output tokens therefore do not represent objects. We introduce an objectness head which predicts the likelihood that an output token actually represents object, and compute boxes, class scores, and losses only for the top k tokens by objectness, similar to Efficient DETR [40]. The objectness head receives an encoder token input and computes scalar objectness score. The objectnesss score predicts the future classification score of a token and is supervised by the actual classification score of those tokens that end up being selected and passed to the classification head. We select approximately 10% of instances by top objectness during training in all of our experiments. During inference, all instances are used.
Mosaies. During self-training, we combine raw images into grids of up to 6 x 6 to produce single training example (i.e. a more extreme version of the mosaics in [26]). This has two main motivations: (1) Using mosaics increases the number of raw images seen for given fixed model input resolution. An alternative is to train using variable image sizes [38], but this would require resizing image position embeddings for each input size. (2) The average resolution and complexity of Web images is lower than images in detection benchmarks and applications. Mosaics reduce the average object size and improve small-object performance, similar to large scale-jittering [8], but with less padding. For all self-training experiments, we use x 1, 2 x 2,3 x 3, 4 x 4, and 6 x grids in equal proportions, resulting in an average of 13.2 raw component images per training example.
To further improve training efficiency, we also adopt previously proposed practices for large-scale Transformer training [42] (details in Appendix A.7). Together, our improvements reduce training FLOPS by approximately 50% compared to the original OWL-ViT [26] and increase training throughput by 2x (e.g. for L/14 at 840 840 resolution measured TPUv3: GFLOPs/example 11'945.4 vs. 5357.9; examples/s/core 1.0 vs. 2.2). We refer to the improved model as OWLv2.
At inference, no token dropping or instance selection is performed. Inference is therefore identical to the original OIT, i.e. each image encoder token is decoded into bounding box and list of per-query classification scores.
5

---

Finc-tuned classes
Unseen classes
"In the Wild" datasets
50
50
ODin mean AP (39
45
40
LVIS Freguee
Pseudo-label space
APar
Curated vocabulary
30
40
3.5
N-grams
2,5
N +curated
85
30
20-
15
30
10'
10a
10
10
10a
10
10'
10
Total examples seen
Total examples seen (including repetitions)
Total examples seen (including repetitions)
(including repetitions)
Figure 2: Comparison of pseudo-label spaces. Self-training on a human-curated list of classes yields good downstream performance on these classes, but generalizes poorly to unseen classes and datasets. Open-vocabulary generalization can be improved by obtaining weak but diverse supervision from image-associated text. WebLI image-text data was pseudo-annotater using OWL-ViT CLIP-L/I4 with one of three label spaces: Curated vocabulary (the union of label spaces from LVIS, Objects365, Openlmagesv4, and Visual Genome), N-grams (lightly filtered N-grams from the text associated with each image), or combination of both (N-grams + curated). self-trained on the pseudo-am -enuttaannttttions and fine-tuned on LVIS Each point represents a separate fine-tuning run. "Examples seen" refers to the number of images after creating mosaics; the total number of raw images seen 13.2x that number (Section 3.2).
WWWLv2-B/10 models were then
3.3 Fine-tuning
Self-training on pscudo-annotations alone already yields strong performance (Section 4.2). However, fine-tuning briefly on human annotations provide significant further benefits. For fine-tuning, we start with the learning rate and optimizer state of the self-trained checkpoint and then continue training on the target dataset while linearly cooling down the learning rate to zero. Fine-tuning of open-vocabulary models involves a trade-off between improving the performance on the fine-tuned classes and losing opca-vocyy performance [30, 29, 37]. We study this trade-off in Section 4.6.
4 Experiments
4.1 Experimental Setup
Models. We use the publicly available OWL-VIT CLIP L/14 model to generate detection pseudo- annotations for the WebLI dataset (10 billion image-text pairs [4]). For all self-training experiments, we use OWL-ViT models modified as described in Section 3.2. Backbones are initialized with the publicly available CLIP [30] checkpoints (B/16 and L/14) or a SigLIP [43] checkpoint (G/14).
Training. Models are first self- on the pseudo- -enett-annotttions for varying durations as indicated. If indicated, after self-training, models are fine-tuned on LVISbase, i.e. the LVIS dataset [10] with all annotations for "rare" categories removed, Therefore, neither the annotator nor any of our models have seen numan-generated annotations for LVIS classes. Fine-tuning uses mosaics up to 3 x 3 and always done until the model has seen 256'000 mosaics (1. (I.IM IM individual images, roughly equivalent to 100 LVIS epochs). The image size is 960 x 960 for /16 models and 1008 x 1008 for /14 models. See Appendix A.8 for a complete list of hyperparameters.
Evaluation. We use mean average precision (mAP) on LVIS [10] as our main detection metric, where mAp rume indicates open-vocabulary performance on unseen classes. To measure generalization on diverse real-world tasks, we evaluate zero-shot performance on the "Object Detection in the Wild" (ODinW) benchmark [21]. ODinW is suite of datasets covering a wide range of domains. We report the average mAP on the subset of 13 ODin datasets introduced in [22] and provide performance on individual datasets in Appendix A.9.2. To avoid leakage of evaluation data into the training set, I was filtered to remove images similar to those in the train, validation, and test splits of 68 common computer vision datasets, including COCOLLVIS, Objects365, and Visual Genome, but not the ODin W datasets (see [4] for details).

---

Fine-tuned classes
Unseen classes
"In the Wild" datasets
45
35
ODin 1.3 mean AP (95
LVIS APPree (%)
45
Confidence
LVIS AP (%)
threshold
30
40
0.1
35
0.3
30
3.5
0.7
15
30
10'
10a
10'
10a
10'
10a
Total examples seen (including repetitions)
Total examples seen (including repetitions)
Total examples seen (including repetitions)
Figure 3: Impact of pseudo-annotation filtering by detection confidence on self-training effectiveness. Pseudo-labels (N-gram label space) were filtered using different confidence thresholds. Number of remaining images for each threshold: 0.1: 5B, 0.3: 2B, 0.5: 782M, 0.7: 224M. OWLv2-B/16 detectors were self-trained on the filtered pseudo-annotations and fine-tuned on LVIS Each point represents a different fine-tuning run. "Examples seen" refers to the number of images after creating mosaics; the total number of raw images seen is 1 that number (Section 3.2).
4.2 Main Result
We compare our best models to the literature in Table 1. We broadly include state-of-the-art open- vocabulary detectors in the comparison. Our self-training approach, using only machine-generatee pseudo-annotation queries, improves over previous methods even without fine-tuning (Table 1, OWL- ST, rows 11-13). Our OWL-ST B/16 model (row 11) achieves 29.6% LVIS mAP points more than the equivalent OWL-ViT model (row 2). Our largest model, G/14 (row 13), reaches 37.5% MAPrare 4.7 points better than the next-best model from the literature (F- (F-VLM R50x64, row 8). Interestingly, after self-training, our models perform better on LVIS mAP rare than mAp (which includes frequent and common classes). We speculate that this may be because weak Web-data supervision may be better for specific terms than general terms: Image/text pairs involving unusual objects (such as L categories) may be more likely to be specifically about these objects, whereas common terms like "person" or "car" may occur often without being related the image.
Fine-tuning on LVIS provides additional significant improvements, even on mAP (OWL- ST+FT, rows 14-16). Our best model, which has only seen machine-generated queries during self-training, reaches 47.2% LVIS mAP mre after fine-tuning, a 14.4 14.4-point improvement over the next best model (F- R50x64, row 8).
Including human--cuatee list of common object classes as seeuo-annotationn queries can further improve the results on LVIS (rows 20-21). but this approach is not fully open-vocabulary since the model sees curated label space, including the LVIS classes, at training time. While the benefit of the curated label space is significant for our smallest model, is is minor on mAP mere for the larger L/14 model (compare rows 15 and 21).
To measure more general open-world performance, Table also includes zero-shot results on ODinW13 [21], a suite of "in the wild" datasets. Performance on ODin is best right after self- training and is reduced by fine-tuning on LVIS We discuss this further in Section 4.6. We also fine-tuned on COCO, where our B/16 and L/14 models reach 54.3% and 56.0% COCO mAP, respectively. OWLv2 therefore matches the performance of ViTDet with Cascade head [23], despite using a simpler head architecture. Further results and examples Appendix A.9.
4.3 Pseudo-Annotation Label Space
Figure 2 takes closer look at the impact of the pseudo-annotation label space on performance after fine-tuning. Performance on fine-tuned classes (mAP left plot) is highest if the pseudo- annotation label space included these classes (blue circles). Therefore, if the target label space is known ahead of time, sneudo-labeling on that space leads to the best results.
However, performance on unseen classes (mAP and "In the Wild" datasets is much better if the pseudo-labeling included diverse queries that were machine- generatdd from the image-associated text (orange squares and green diamonds). A mixture of human and machine- machine-generated label spaces
7

---

Fine-tuned classes
Unseen classes
"In the Wild" datasets
50
50
ODinW13 mecn AP (%)
LVIS mrpppe (%)
45
45
Model
LVIS AP... APam
OWL-ST+FT B/16
40
OWL-ST+FT L/14
x
x LW14 (annotator)
35
30
30.1
101
10
10
10a
10
Detection training compute
Detection training compute (exaFLOPs)
Detection training compute (exaFLOPs)
(exaFLOPs)
Figure 4: Scaling of detection performance with model size and training compute. Models show classic scaling behavior [42]: Performance increases monotonically with training compute, with larger models being necessary to benefit from larger amounts of compute/data. Models were self-trained on N-gram pseudo-annotations and fine-tuned on LVIS
performs well in all settings, but does not significantly outperform the purely machine-generated label space on the "In the Wild" datasets. These results suggest that a human-curated label space can help if the target label space is known, but that strong in-the-wild generalization driven by the weakly supervised machine-genen machine-generated label space. Our results also show that a simple N-grams approach is sufficient leverage the weak supervision and outperforms more complex methods (Table 1).
4.4 Filtering of -seudo-Annotatioss
Besides the label space, a second important decision self-training the filtering of pseudo- annotations. We filter based on the detection confidence score of the annotator and vary the score threshold in Figure 3. For confidence-basee filtering, bias-variance trade-off exists between including only nigh-confidence neuuo-annotations but inheriting the annotator's biases, or lowering the bias but increasing the noise by including lower-confidence works err the side of high bias and low variance, applying high confidence thresholds [35] or including only the single highest-confident that including all secudo-annotations that pass moderate threshold of 0.3 works well, while strict thresholds lead to poor results (Figure 3). As training continues for longer than what was possible for Figure 3, suspect that lower thresholds may scale better. Therefore, for our main results, chose to include all annotations above 1, but only kept images with at least one annotation above 0.3.
cecudo-annotations Many prior
detection for an image [47, 1]. In our setting, we find
4.5 Scaling
The use of abundant Web image-text data with little filtering means that our self-training dataset large (approximately 2B images). We can therefore study detection training scaling in the same regime as prior work on classification (Figure 4; models see each image at most once for these experiments). We make several noteworthy observations:
1. Self-training is beneficial already at moderate compute budgets, less than that of the annotator.
Models show similar scaling behavior for detection as for classification [42]: Both overall performance and the size of the Pareto-optimal model increase with compute/data size.
As we move further out distribution, the amount of compute which L/14 overtakes B/16 increases. In other words, for in-the-wild performance, at most compute budgets, it may be better to train a smaller model for longer than larger model for shorter.
These results suggests that self self- training on Web data is further scalable as an approach for improving open- vocabulary localization models without the need for further human annotations. The large datasets also makes it possible to scale model size. We trained a G/14 model, which has 5.2 the number of parameters and 4.3x the inference FLOPs of our L/14 model. To our knowledge, this the largest open-vocabulary detection model to date. Since the G/14 model uses a different backbone than our other models (SP [43] instead CLIP [30],, we not include in Figure 4, but show in Table that it is currently the best-performing model zero-shot LVIS, with 47.2% mAp rare
8

---

55
Ensopbies
LVIS APrequmn (%)
50-
Tomterrr fine-tomin tonting
20k
Sk
24
LVIS AP (%
20K
Self-training only
3
Fine-tuned on LVIS base
x
100
2
Weight ensemble
30
OWL L/14 (anmotator)
Longer seal--raining
25
25
44
46
48
50
52
54
44
46
48
50
52
54
"In the Wild" performance
"In the Wild" performance
(0DinW13 mean AP (%))
(ODinW13 mean AP (96)
Figure 5: Trade-off between fine-tuned and open- open-world performance. Self-training yields continued improvements on suite of diverse datasets (ODinW133 13; zr-aii), but performance on any given dataset (e.g. LVIS; y-axis) may saturate (red circles). Fine-tuning on target dataset improves performance on that dataset, but reduces the open-world generalization ability in proportion to the finetuning duration (light blue squares; numbers indicate finetuning steps). This trade-off can be improved through weight-space ensembling (averaging) of the pretrained and fine-tuned checkpoints [37] (purple diamonds; numbers indicate the mixing coefficient for the fine-tuned weights). The plot shows B/16 models self-trained on N-gram useudo-annotation training or after fine-tune tine-tuning on LVIS Ensembles were created between the longest-self- trained checkpoint and the weights obtained after finetuning that checkpoint for 20k steps. Note that there is significant variability in ODinW13 performance between checkpoints towards the end of self-training.
and evaluated either directly after self-
4.6 Effect of Fine- Tuning on Bren-Vocabulary Performance
For contrastively trained image-text models, fine-tuning improves performance the target distribu- tion but reduces the (originally very high) robustness to distribution shift [30, 28, 37J. We observe the same effect for detection, using ODin 13 AP proxy for out-of-distribution performance: Compared to the performance after self self-training (red dots in Figure 5), fine-tuning on LVIS im- OOD performance (ODinW13 AP) is simultaneously reduced in proportion to the amount of fine-tuning (light blue line in Figure 5).
proves performance on the fine-tuned classes (LVIS MA Pirqquen)))
A simple approach to improve this rofe-oof'se is to create ensemble of the model before and after fine-tuning by averaging the model weights [37]. This approach comes at no additional training cost and improves the Pareto-frontier for all ensemble mixing ratios (Figure 5, purple line). We also tried co-training on WebLI and LVIS but found it to perform worse than weight ensembling.
Notably, performance LVIS behaves similarly to LVIStrequent and improves during fine-tuning, even though no LVIS classes are seen (Figure 5, right). This may be because LVIS classes semantically and visually close to frequent classes. For example, seeing many annotations for "bird" may improve performance on rare classes such as "heron". "mallard", or "puffin". LVIS mAP ture therefore only measures narrow concept of open-vocabulary performance, and does not reveal the fact that fine-tuning significantly reduces generalization to broader distribution shifts. Benchmarks such WDinW therefore provide significant additional insight.

---

Limitations
The main limitation of our method is the amount of compute and data needed for self-training. As we show in Section 4.5, performance improves consistently with training compute and data. This means that further improvements are possible, but also that these will come increasingly large costs. In fact, cost likely increases faster than resources can realistically be grown in practice. New approaches will therefore be eventually necessary for further improvements.
A second important limitation of our method, similar to other open- vocabulary models [30. 28, 37], the trade-off between fine-tuned and open-vocabulaty performance addressed Section 4.6. For out-of-distrinn of-distribution queries, predictions of fine-tuned models may be poorly calibrated and may depend on the precise wording of the query. These issues can be mitigated with weight ensembling [37], but more research is needed to fully understand the vee--ocabullay robustness of these models.
6 Conclusion
In the past, open-vocabulary detection performance has been limited by the availability of human- annotated detection training data. Here, we show that self-training can be scaled up to overcome the dependency on human annotations, Our OWL-ST recipe delivers large improvements in detection performance using weak supervision from abundant Web data, similar what has been seen for image classification and language modelling.
Acknowledgments and Disclosure of F Funding
We would like to thank Xiao Wang for help with the WebLI dataset, Xiaohua Zhai and Lucas Beyer for providing the SigLIP model, and Rich Munoz and Alexey Dosovitskiy for insightful comments.
References
[1] Arandjelović. R., Andonian, A., Mensch, A., Hénaff, OJ.,, Alayrac. J.B.. Zisserman, A.: Three ways to improve feature alignment for open vocabulary detection. arXiv preprint arXiv:2303.13511 (2023)
[2] Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. arXiv preprint arXiv:2210.0946) (2022)
[3] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M.J., Leary, C., Maclaurin, D., Necula, G., Paszke, A.,
VanderPlas, J., Wanderman-Milne, S., Zhang, Q:: JAX: composable transformations of Python+NumPy
programs (2018), http:: http:/ttuuuucooooooo.egeoo com/google/jax
[4] Chen, X., Wang, X., Changpinyo, S., Picrgiovanni, A., Padlewski, P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigeerver, J., Ding. N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A. Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B.K., Ruiz, C.R., Steiner, A.P., Angelova, A., Zhai, X., Houlsby, N., Soricut, R.: PaLI: jointly-scaled multilingual language-image
model. ICLR (2023)
[5] Dave, A., Dollár, P., Ramanan, D., Kirillov, A., Girshick, R.: Evaluating large-wocabulary object detectors: The devil is in the details. arXiv preprint arXiv:2102.01066 (2021)
[6] Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heck, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Riquelme, C.. Minderer, M., Puigcerver, J.. Evei, U., Kumar, M., van Steenkiste. S., Elsayed, G.F., Mahendran, A., Yu, F. Oliver, A., Huot, F., Bastings, J., Collier, M.P., Gritsenko, A., Birodkar, V., Vasconcelos, C., Tay, Y.. Mensink, T., Kolesnikov, A., Pavetić, F,, Tran, D., Kipf, T., Lučić, M., Zhai, X., Keysers, D., Harmsen, J., Houlsby, N.: Scaling vision transformers to 22 billion parameters. ICML (2023)
[7] Dehghani, M., Gritsenko, A.A., Arnah, A., Minderer, M., Tay, Y.: SCENIC: A JAX library for computer vision research and beyond. arXiv preprint arXiv:2110. 1103 (2021)
[8] Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.Y., Cubuk, E.D., Le, Q.V., Zoph, B.: Simple copy-paste strong data augmentation method for instance segmentation. CVPR (2021)
[9] Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: bpen-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921 (2021)
[10] Gupta, A., Dollar, P., Girshick, R.: LVIS: A dataset for large vocabulary instance segmentation. CVPR (2019)
10

---

[11] Heek, J., Levskaya, A.. Oliver. A., Ritter, M., Rondepierre, B., Steiner, A.. van Zee, M.: Flax: A neural network library and ecosystem for JAX (2020), http:// cow/googlee sagooggleffoo flax
[12] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q. Sung, Y.H. Li, Z., Duerig, T: Scaling up visual and vision-language representation learning with noisy text supervision. ICML (2021)
[13] Jouppi, N.P., Yoon, D.H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., Patterson, D.: A domain- specific supercomputer for training deep neural networks. Communications of the ACM 63(7), 67-78 (2020)
[14] Kamath, A., Singh, M., LeCun, Y., Synnacve, G. Misra, I., Carion, N.: MDETR modulated detection for end-to-end multi modal understanding. ICCV (2021)
[15] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arviv preprint arXiv:1412.66: (2014)
[16] Kirillov, A., Mintun, E.,, Ravi, N., Mao, H., Rolland, C.,, Gustafson, L.,, Xiao, T. Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv preprint arXiv:2304.0264 (2023)
[17] Kolesnikov, A.. Dosovitskiy. A., Weissenborn, D., Heigold, G., Uszkoreit, J., Beyer, L.. Minderer, M.. Dehghani, M.. Houlsby, N., Gelly, S., Unterthiner, T., Zhai, X.: An image is worth 16x16 words: Transformers for image recognition at scale. ICTR (2021)
[18] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, LJJ,, Shamma, D.A., et al: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123(1), 32-73 (2017)
[19] Kuo, W., Cui, Y. Gu, X., Piergiovanni, A., Angelova, A.: Open open-vocabulary object detection upon frozen vision and language models. ICLR (2023)
[20] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, Kamali, S., Popov, S., Malloci, M.. Kolesnikov, A., Duerig. T., Ferrari, V.: The Open Images Dataset V4. International Journal of Computer Vision 128(7), 1956-1981 (Mar 2020)
[21] Li*, C., Liu", H., Li. L.H., Zhang, P., Aneja, J., Yang. J.. Jin, P., Lee, YJ.. Hu. H., Liu, Z., et al.: Elevater: A benchmark and toolkit for evaluating language-augment augmented visual models. NeurIPS (2022)
[22] Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L.., Yuan, L.., Zhang, L.., Hwang, J.N., et al.: Grounded langu pre-training. arXiv preprint (xXiv:2112.0356 (2021)
[23] Y., Mao, H., Girshick, R., He, K.: Exploring plain vision transformer backbones for object detection.
arXiv preprint WtXiv:220311652 (2022)
[24] Liang, Y. Ge, C., Tong, 7., Song, Y., Wang, J., Xie, P.: Not all patches are what you need: Expediting vision transformers via token reorganizations. ICLR (2022)
[25] Marin, D., Chang, J.H.R., Ranjan, A., Prabhu, A., Rastegari, M., Tuzel, O.: Token pooling in vision transformers for image classification. CVPR (2023)
[26] Minderer, M., Gritsenko. A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A.. Amab, A., Dehghani, M., Shen, Z., Wang. X., Zhai, X., Kipf, T., Houlsby, N.: Simple veceb-locy object detection with vision transformers. ECCV (2022)
[27] Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, L.D., Gebru, T.: Model cards for model reporting. In: Proceedings of the conference on fairness, accountability, and transparency. pp. 220-229 (2019)
[28] Pham, H., Dai, Z.. Ghiasi, G. Kawaguchi, K., Liu, H., Yu, A.W., Yu, J., Chen, Y.T., Luong, M.T., Wu, Y..
Tan, M., Le, Q.V.: Combined scaling for zero-shot transfer learning (2021)
[29] Pham, H., Dai, Z., Ghiasi, G., Liu, H., Yu, A.W., Luong, M.T., Tan, M., Lc, Q.V.: Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111 1..15000 (2021)
[30] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Gob, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P.,
Clark, J., Krueger, G., Sutskever, L:: Learning transferable visual models from natural language supervision. ICML (2021)
[31] Ramanathan, V., Wang, R., Mahajan, D.: Dlwi: Improving detection for lowshot classes with weakly labelled data. CVPR (2020)
[32] Rao, Y., Zhao, W., Liu, B., Lu, J.. Zhou, J.. Hsieh, C.J.: Dynamievit: Efficient vision transformers with dynamic token sparsification. NeurIPS (2021)
[33] Shao, S., Li, 7., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.: Objects365: A Large-Scale, High-Quality Dataset for Object Detection. ICCV (2019)
[34] Shazeer, N., Stern, M.: Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04233 (2018)
[35] Sohn, K., Zhang. Z., Li. C.L., Zhang, H., Lee, C.Y., Pfister. T.: A simple semi-supervised learning framework for object detection, arXiv preprint 2XXv:22005.04777 (2020)
11

---

[36]
Vaswani, A., Shazeer. N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.,, Polosukhin, I:: Attention is all you need. NeurIPS (2017)
[37] Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Gontijo-Lopes, R., Hajishirzi, H.,
Farhadi, A., Namkoong. H., Schmidt, L.: Robust fine- tuning of zero-shot models. CVPR (2022)
[38] Yao, L., Han, J., Liang, X., D., Zhang, W. detection pre-training via word-region alignment. arXiv preprint arXiv:2304.0493 (2023)
Xu, Detclipv2: Scalable open- open-vocabulat object
[39] Yao, L.,, Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., Xu, H.: DeCLLIP: Dictionary- enriched visual-conceny paralleled pre-training open- detection. NeurIPS (2022)
[40] Yao, Z., Ai, J., Li. B., Zhang, C.: Efficient detr: Improving end-to-ena eod object detector with dense prior.
arXiv preprint arXiv:2 (2021)
[41] Yin, H., Vahdat, A., Alvarez, J., Mallya, A., Kautz, J., Molchanov, P.: Adavit: Adaptive tokens for efficient vision transformer. CVPR (2022)
[42] Zhai, X., Kolesnikov, A., Houlsby, N., Beyer, L.: Scaling vision transformers. CVPR (2022)
[43] Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. arXiv
preprint arXiv:2303.1334 (2023)
[44] Zhai, X., Wang. X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: LiT: Zero-shot
transfer with locked-image text tuning. arXiv preprint arXiv:2111 (2021)
[45] Zhang. H., Zhang, P., Hu, X., Chen, Y.C., Li, L.H.,, Dai, X., Wang. L.,, Yuan, L.., Hwang, J.N., Gao, J.: Glipv2: Unifying localization and vision-language understanding. NeurlPS (2022)
[46] Zhong, Y. Yang. J., Zhang, P., Li, C., Codella, N., Li, I.H.., Zhou, I.., Dai, X., Yuan, I.,, Li, Y., et al.:
RegionCLIP: Region-baset based language-image pretraining. arXiv preprint arXiv:21 (2021)
[47] Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., Misra, I.: Detecting twenty-thousand classes using
image-level supervision. arXiv preprint arXiv:2201.022 (2021)
[48] Zoph, B., Ghiasi, G., Lin, T.Y., Cui, Y. Liu, H., Cubuk, E.D., Le, Q.V.: Rethinking pre-training and self self-training. NeurlPS (2020)
12

---

A
Appendix
The Appendix provides a Model Card [27] for OWLv2 as well as additional methodological details, hyperparameters, and results. At the end of the Appendix, we provide qualitative examples of the self-training data and model predictions.
The Appendix is structured as follows:
A.1 Model Card
14
A.2
Human-Curated Label Space
15
A.3 Machine-Generatel Label Space
15
A.4 Combined Label Space
16
A.5 Augmentations for Self-Training
16
A.6 Token Dropping....
17
A.7 Further Efficiency Improvements
18
A.8 Model Hyperparameters
18
A.9 Additional Results
19
A.9.1 Fixed Average Precision
19
A.9.2
Per-Dataset W Results
19
A.9.3
Fine- Tuning Robustness Trade- Off for OWLv2 L/14
19
A.10
Qualitative Examples
19
13

---

A.1
Model Card
Model Summary
Model Architecture
OWL v2 is an opca-voca object detector based on OWL-ViT [26]. It consists of an image encoder with a Vision Transformer [17] architecture, text encoder with similar Transformer architecture, and heads that predict bounding boxes and label scores from provided images and text queries.
Input(s)
An image and list of free-text object descriptions (queries).
Output(s)
A list of bounding boxes and score for each box/query pair.
Usage
Application
The model is intended for open voca-locay object detection.
Known Caveats
(1) Confidence scores of predictions are not intended to he compared across text queries. While the training loss encourages cross-query calibration for seen queries, scores for unseen queries not calibrated. Further, the mean Average Precision (mAP) metric does not measure cross-query calibration. so higher mAP does not imply better cross-query calibration. Also see Section 5. (2) Fine-tuning the model creates a tradc-off between the performance on fine- tuned texts and unseen texts. See Section 4.6 for details.
System Type
System Description
This is a standalone model.
Upstr. Dependencics
None.
Downstr. Dependencies
None.
Implementation Frameworks
Hardware & Software
Hardware: TPU [13] v2 or v3 (for B- and L-sized models) v4 (for G-sized
models). Software: JAX [3], Flax [11]. Scenie [7].
Compute Requirements
Reported in Section 4.5.
Model Characteristics
Model Initialization
The model is initialized from pre-trained language CLIP [30] SigLIP [43] checkpoints.
Model Status
This is static model trained an offline dataset.
Model Stats
The largest OWLv2 model has 2.3B parameters, of which 2B are used for the image encoder and 300M for the text encoder (the heads have negligible number of parameters). We also trained models with 430M and 150M parameters.
Data Overview
Training dataset
The model is self-trained on bounding boxes predicted by the original OWL-VIT L/14 model [26] the WebLI dataset [4]. Details the annotation procedure
are provided in Section 3.
Evaluation & Fine-tuning Dataset
Open bven-vocabulary object detection performance is evaluated using the LVIS [10] and ODinW13 [21] datasets.
indicated Table 1, some models are fine-tuned on the "base" annotations of LVIS, i.c. only annotations for "frequent" and "common" object categories as defined in the official annotations [10]. None of our models have seen any human annotations for IVIIS "rare" categories, such that LVIS mAPare measures zero-shot performance.
Evaluation Results
Evaluation Results
Reported in Table 1.
Model Usage & Limitations
Sensitive Use
The model detects objects matching free-text descriptions. This capability should not be used for unethical use cases such as surveillance.
Known I Limitations
Reported in Section 5.
Ethical Considerations
Reported in Section
14

---

A.2 Human- Human-Curated Label Space
The human- cumanne label space was obtained by merging common dataset class lists with the Python code below.
Dataset class names, as available e.g. from TensorFlow Datasets. For Visual Genome, we used the 1600 most common label strings. LVIS CLASS NAMES [...]
0BJECTS365 CLASS NAMES [...]
OPEN IMAGES V4 BOXABLE CLASS NAMES [...]
VISUAL GENOME WAAS__NAEES [...]
queries
LVIS CLASS NAMES
10
0BJECTS365 CLASS NAMES
11
OPEN IMAGES V4 BOXABLE CLASS NAMES
12
Remove duplicates:
15 queries set (lq. lower() for q in queries])
16
17 Remove plural forms:
remove s)
19 for singular queries:
20 plurals = [singular + 's', singular 'es')
21 for plural in plurals:
if plural in queries:
23
aemove.add
24
25 Same queries for all images:
queries list(queries dinueriie.d.feeeenccereeeeo
A.3
Machine-Generated Label Space
The machine machine-generated label space obtained from the image-associated text, for each image separately, using the Python code below. Figure A3 shows example pseudo-annotations using the N-gram label space.
from typing import Iterable, List import nltk
Stopwords from nltk. www.ssoorru....oo.co wwwpus..soppooooo..o.o www.sooooo..ooreeoooss ('englis"")) ): STOPWORDS EN frozenset({
'a', 'about', "above', 'after', 'again', 'against', 'all', 'am', 'an', 'and', 'any', 'are', 'as", 'at', 'be', 'because', 'been', 'before', 'being', 'below', 'between', 'both', 'but', 'by', 'can', 'did', 'do', 'does' 'doing', 'don' down' 'dring'', 'each', 'few', 'for', 'from' 'further', 'had', 'has', 'have', 'having', 'he', 'her', 'here', 'hers', 'herself", 'him', 'himself', 'his', 'how', 'i', 'if', 'in', 'into', 'is' 'it', 'its', 'itself', 'just', me' 'more', 'most', 'my', 'myself', 'no', 'nor', 'not', 'now'. 'of', 'off', 'on', 'once', 'only' 'or', 'other' 'our', 'ours' 'ourselves', 'out', 'over', 'own', 's', 'same', 'he'' 'should', 's0', "some", 'such', 't', 'than', 'that', 'the', 'their', 'thiirs', 'them" 'themselves', 'then', 'there', 'these', 'they', 'tii', 'those', 'through', 'to', 'too', 'under', 'until', 'up', 'very', 'was', 'we', 'were', 'what', 'when', 'where', 'which', 'while', 'who', 'whom', 'why', 'will', 'with', 'you', 'your', 'yours', 'yourself', "yourselves"
10
13
14
17
18
19
20 })
21
22 These words were found by manually going through the most common 1000 words
23 in a sample of texts and selecting generic words without specific meaning:
24 COMMON GENERIC WORDS frozenset({ 'alibaba', 'aliexpress', 'amazon', 'available', 'background', 'blog', 'buy',
15

---

26
'co', "com', 'description', "diy', 'download', 'facebook; 'free', "gif", 'hd', 'ideas', 'illustration' 'illustrations,, 'image', 'images', 'ing'. 'instagram' 'jpg', 'online', 'org', 'original', "page', 'pdf', 'photo', "photography', "photos', "picelick', 'picture', 'pictures', 'png', 'porn' "premium', 'resolution', 'royalty', "sale', "sex', "shutterstock', 'stock', 'svg', 'thumbnail', 'tumblr', 'tumgir', 'twitter', 'uk', 'uploaded', 'vector', 'vectors', 'video', 'videas', 'wallpaper', 'wallpapers', 'wholesale', 'www', 'xxx', 'youtube"
27
28
29
30
33
34 }
35
def is all stopwords(ngram: Iterable[str] bool: return set(ngram) iSSUUse
37
38
39
40
def @get_ngramst
41
caption: str, queries: int, max ngram len: int List[str]:
42
43
"""Returns image caption ngrams as queries."
44
Make lower-case:
46
caption caption. lower()
47
48
Remove common generic words:
words
[w for in split) if w not in COMMON GENERIC [ORDS]
51
queries []
52
for ngram in nvtk.everygrams (words, max len=max ngram len):
53
# Don't use ngram if it only consists of stop words:
if __is_aa si__al_ssoopull (nnglo continue
55
56
queries. aperede (.join(nrram))
57
1f Len (queries) == max num queries:
break
return queries
60
61 Example command to get queries for one image:
62
queries _get wannggram (caption, max num hww_unuuuuuu___ee 300, max_ _ora_ngraa___ len=1e))
A.4 Combined Label Space
When merging neuudoannotations obtained with human- curated and machine machine-generated queries, it important to consider that human-curated queries tend to be closer to the training distribution of the annotator and therefore tend to have higher scores than escudo-annotation based machine- generated queries. Simply merging annotations from the two label spaces and filtering them with the same confidence threshold would therefore retain primarily annotations based on human-curated queries. To achieve a more even balance when using the combined label space ("N- "IN-gmmccurated in Table 1), therefore re-scaled scores of pseudo secudo-nnntttionns obtained with the human-curated queries by a factor of 0.3 before applying the same confidence threshold to all (human (human-curated curated and machine-generated) annotations.
A.5 Augmentations for Self- Self-Training
Since Web-scale image-text data differs in important aspects from human-curated detection datasets, we depart from the augmentation strategy of [26] in several ways. As described in Section 3.2, since Web images tend to be smaller and show fewer objects than e.g. LVIS images, we use stronger image mosaics with up do 6 6 tiles (Figure A1). For the same reason, additionally randomly resize each raw image such that its width between 0.5x x and 1.0x the width of the full mosaic tile, padding on the bottom and right preserve the aspect ratio (Figure A4).
On the other hand, given the large size of our dataset, some other augmentations can be avoided: We do not use left/right flipping random cropping during self-training. We also do not add random
16

---

All
Frequent
Rare
30
LIVIS AP fropest (%)
LVIS APai (3)
LVIS AP
26
12x12
12x12
3x3
12x
Max. mosaic size
Max. mossic size
Max. mosaic size
Large
Medium
Small
LVIS AP mut (5)
LVIS APorg
LVISS medun
48
33-
15
3x3
6x6 8x8
12x12
3x3
6x6 8x8
12x12
3x3
6x6
8x8
12x 12
Max. mosuic size
Max. mosaic size
Max. mosuic size
Figure A1: Sweep over mosaic sizes. OWL-ViT B/16 models were trained on pseudo-box annotations ("ngrams" label space) for 100'000 steps with different mosaic sizes. At given "max. mosaic size", the model is trained on equal proportions of mosaics up to that size. For example, for max. size 12 12, the model receives images with 1, 2 32 42, 62, 82, 122 tiles, respectively (only sizes with prime factors 1, 2, and are supported). For this figure, the model input resolution was 768 768. Mosaic sizes up 12 12 improve overall performance (mAPan) and especially "rare" and "small" object performance. The benefit may be due to seeing smaller objects on average, or due seeing more WebLI images per training step (a x 12 mosaic contains 144 WebLI images).
prompt templates to the pseudo-label, during self-training. During fine-tuning, we use the same augmentations as [26].
A.6 Token Dropping
To improve training efficiency, we drop image patches based on their pixel variance (Section 3.2). Table A2 shows how the performance of standard OWI-VVTT model varies for different amounts of token dropping- Dropping up to 50% of tokens within one standard deviation of the full performance. We therefore drop 50% of tokens during all of our experiments.
Table A2: Performance of standard OWL-VIT (L/14), trained on Objects365 and Visual Genome as in [26], for different token drop rates. For drop 0.0, the standard deviation over three given.
Token drop rate
Metric
0.00
0.25
0.33
0.50
0.70
LVIS Ap
33.3 +0.33
33.1
33.6
32.9
30.4
LVIS Aptal
31.8 #1.16
31.0
32.6
30.8
28.2
To inject some stochasticity to the patch selection, we add small amount of noise to the image before computing patch variance (uniformly distributed between 0.0 and 0.01 for images the range [0.0, L.0)). Figure A4 shows example training image before and after token dropping.
17

---

Table A3: Hyperparameters of the models shown in Table 1. Only parameters that vary between models are shown; constant parameters are described the text (Appendix A.8). For Dropout rate and Droplayer rate, the first number indicates the value used for the image encoder and the second for the text encoder. Examples seen includes both self traininn and fine-tuning.
Droplayer rate
Batch size (ST)
Batch size (FT)
Examples seen
Learming rate
Dropout rate
Instance top
Image size
Method
Backbone
Open vocabulary:
11 OWL-ST
CLIP B/16
960
5
10-3
.0.0
.2.11
256
256
3.7 x 10
12 OWL-ST
CLIP L/14
1008
2 10
.0.0
2/11
512
256
2.3 x 10
13 OWL-ST
SigLIP G/14
1008
2 10
.0.1
.2.4
512
128
1.6 x 10
14 OWL-ST+FT
CLIP B/16
960
x 10
.0.0
2/.1
256
256
256
3.6 x 10
15 OWL-ST+FT
CLIP L/14
1008
2 10
.0.0
2/11
512
256
128
2.3
10
16 OWL-ST+FT SigLIP G/14
1008
2 10
.0/1
2/.4
512
128
128
1.6 x 10
Human-curated vocabulary:
20 OWL-ST+FT
CLIP B/16
960
5
10
.0/0
2/.1
256
256
256
8.2
10
21 OWL-ST+FT CLIP 1/14
1008
2 x 10
.0/0
2/11
512
256
128
3.6 x 10
A.7 Further Efficiency Improvements
To further improve training efficiency beyond the methods deseribed in Section 3.2, we also adopt previously proposed methods for large-scale Transformer training: To save memory, we use variant [42] the Adafactor optimizer [34] instead of Adam [15]. To avoid having to choose and optimize the total training duration ahead of time, we use the open-ended inverse square-root schedule [36, 42] with a fixed time-scale of 10'000 steps for all experiments and linearly "cool down" checkpoints along the way for evaluation (see Section 3.3).
A.8 Model Hyperparameters
We use the following hyperparameters for all of our models. Hyperparameters that vary between models are listed in Table A3.
Optimizer: Adafactor variant as in [42]
Learning rate schedule: Inverse square-root [36] with timescale 10'000 steps
Learning rate for the text encoder: 2 x 10
Token dropping rate during training: 0.5
Pseudo-annotation confidence score threshold: 0.3 (except for Figure 3)
Augmentations: See Appendix A.5
All remaining hyperparameters are as in [26].
Hyperparameter selection. Most hyperparameters were either taken directly from [26] or tech- nically constrained, e.g. chose the largest batch size that fit into the memory of the available accelerators. Where hyperparameters were tuned, we ran short B/16-scale trial experiments and selected the parameters with the highest LVIS mAP fare for our main runs.
SigLIP G/14. For the G/14 model, we started self-training with a learning rate of 10-5 droplayer rate of 1/.0, and no dropout. We found that the model overfit during fine-tuning with these settings, and switched to a learning rate of 10-5 a droplayer rate of .2.44 and dropout rate of .0/.1 after 740'000 self-training steps. To save resources, we did not start training from the beginning. With the new settings, we observed no overfitting during fine-tuning, but it is possible that these settings are still not optimal.
18

---

Table A4: Open-vocabulary detection results on LVIS using the "fixed" AP metric [5]. Fixed AP is implemented as proposed in [5] by evaluating AP on the top 10'000 predictions per class over the entire validation set.
Apmmni
A pamini
AP fixed
A A
Method
Backbone
fixed old fixed
old
fixed
Open vocabulary:
RegionCLIP [46]
R50x4
32.3
22.0
2
OWL [26]
CLIP B/16
27.2
20.6
3
OWL [26]
CLIP L/14
34.6
31.2
4
GLIPv2 [45]
Swin-T
29.0
GLIPv2 [45]
Swin-B
6
GLIPv2 [45]
Swin-H
50.1
7
F-VLM [19]
R50x4
28.5
26.3
8 F-VLM [19]
R50x64
34.9
32.8
3Ways [1]
NFNet-FO
35.7
25.6
3Ways [1]
NFNet-F6
44.6
30.1
11 OWL-ST
CLIP B/16
31.8
34.4
35.4
38.3
27.0
28.6
29.6
30.3
12 OWL-ST
CLIP L/14
38.1
40.9
39.0
41.5
33.5
35.2
34.9
36.2
13 OWL-ST
SigLIP G/14
37.8
40.9
33.7
37.5
14 OWL-ST+FT
CLIP B/16
47.2
48.7
37.8
42.1
41.8
43.2
36.2
39.0
OWL-ST+FT
CLIP L/1
54.1
56.2
46.1
52.3
49.4
51.1
44.6
47.4
16 OWL-ST+FT
SigLIP G/14
51.3
50.9
47.0
47.2
Human-curated vocabulary:
17 Detic [47]
R50
32.4
24.6
18 DeICLIPy2 [38]
Swin-T
40.4
36.0
32.8
31.0
19 DeICLIPv2 [38]
Swin-I.
44.7
43.1
36.6
33.3
20 OWL-ST+FT
CLIP B/16
51.1
52.3
41.9
46.5
45.6
46.7
40.5
42.5
21 OWL-ST+FT
CLIP L/14
55.8
57.2
50.0
54.5
50.4
52.0
45.9
48.5
A.9 Additional Results
A.9.1 Fixed Average Precision
In the standard Average Precision metric (Apold), performance one class depends on the perfor- mance on other classes. This dependence makes the metric "gameable" by re-scaling the scores of certain classes [5]. To avoid this issue, some prior work reports "fixed" version of AP proposed in [5]. In Table 1, we report A for our models. For models from the literature, we report whichever AP version is available. Since Apfixed tends to produce higher values than Apald Table tends to underestimate the advantage of our method over prior work using Apfixed We provide A pliised for all of our models in Table A4. As proposed in [5]. we implement Aplixed by evaluating AP on the top 100000 predictions per class over the entire validation set. This ensures that classes do not compete with each other for inclusion in the evaluated predictions.
A.9.2 Per-Dataset WDin Results
Table A5 shows un-aggregated results all 35 ODinW datasets for main models. In addition, in the last row, we provide results for a weight-space ensemble of self-trained and fine-tuned OWLv2 L/14 model (the same model is shown in Figure A2).
A.9.3 Fine-Tuning Robustness Trade-Off for OWLx2 L/14
In Figure A2, we provide the same analysis of the robustness trade-off after fine-tuning for an L/14 model that provided for B/16 model in Figure 5.
A.10 Qualitative Examples
In Figures A5 to A7, we provide qualitative examples of detection predictions from OWLv2 L/14 models. In each figure, the top image shows predictions obtained directly after self-training, and
19

---

the bottom image shows predictions after fine-tuning on bViS Example images are from the LVIS validation set and the model was queried with all LVIS classes. All predictions meeting the confidence threshold specified in the caption are shown.
55
LVIS APrequem (%)
L.VIS Arrure (%)
3
Self-training only
40-
Fine-tuned on L.VIS base
100
40-
Weight ensemble
35
OWL L/14 (annotator)
x
30
44
46
48
50
52
54
56
44
46
48
50
52
54
56
58
"In the Wild" performance
"In the Wild" performance
(ODinW13 mean AP (%))
(ODinW13 mean AP (%)
Figure A2: Trade-off between fine-tuned and open- world performance. Similar to Figure 5, but for OWLv2 L/14.
Table A5: sero-shot AP of the models Table on 35 ODinW datasets [21]. The subset of 13 datasets defined in [22] and used in the main paper is shown in bold. The last row (OWL-ST/FT ens) shows the weight-space ensemble [37] of the checkpoints after self-training and after fine-tuning of the model in row 21 (weight of the fine-tuned checkpoint in the ensemble is 0.4; also Figure A2). This is our best model by ODinWI performance.
Acrial Mariiinc Drose Large
Actial Martine Drone 11l0d
Mountai Dee Commerc af
North America Mushhroess
Thoraal Dags And People
Moin dataset))
05 dansess)
American Langrage
Boggle Boans
Bookke Undorwater
Choss Ploces
Contontal Rahhie
Doe Medium Color
Dooms Contall
Ego Generie
Ego Hords Syecitie
Weekers
Mask Weuring
Vision
Ortordd sy
Outtrd Pas. By Specios
Selibbiving Car
Shutllish Opentmaggs
Thermal Chestah
Vohiees Opentmage Images
Wildtir Stuake
Aquariam
BCCD
Puckages
Pk Lot
Plandoe
Fothole
Raccmon
Open vocabolory: OWT.ST
48
33.
0.1
19.1
OWL-ST
C/14
53,0
24.4
16.2
199
21.2
1.1
32,5
16.2
02
3.9
7.8
0.1
4.7
47.1
3.5
22.3
76.
0.6
3.1
2.7
10.9
53.9
62.6
0.0
4.4
27.5
63.8
49
35.0
25.5
55.6
58.5
31.1
OWL-ST
SipLIP G/14
48.9
17.5
22.0
17.5
36.7
21.4
3.3
5.6
8R.
0.1
4.9
37.8
4.3
22.6
42.
0.5
3.0
322
62.8
53.4
0.1
6.5
25.7
83
5.8
42.5
25.
56.5
58.1
23
16.3
0.1
47.7
OWL-STIFT
SigLiP Go14
21.3
16.5
5.6
S.
25.5
077
23.6
47.9
619
Human-corated secabulary:
OWL
CLIP
16.7
172
35.3
4.6
4.4
0.1
27.3
04
62.6
20.6
OWI-ST-FT
CL141/14
63.2
OWL-TIFF
CLIP1/14
20

---

file 2005 citroën c3 interior ath july 2015...
map athenian empire svg
file bus luxembourr city 3 jpg
file patrick henry bruce landscape google art...
cltrodn mondee
TE mepp otheran
nnturtrr
453 DC 41
e3 Interior
the dital bus
431 be cl
datal bus
Rie map achen
file monumento josé maria das santos pinhal...
file fresco showing a silver tray containing...
file iss 39 grand canyon jPg
file kalisz bazylika
wniebowzi
cia nmp...
monament
driefiguing dried
monument
Sipuunns dried
file station tramway ligne 3b marie miribel...
file gleisanlage zürich hauptbahnhof jpg
fichier h with hook uppercase and lowercase...
archivo red script palestine flag svg
Camii nock uppertase taee
track ayaatum suriic manns
Trark mur mein wiselon
Brack systemum
track ayabem
track systemy
datei kitchen impossible map png
gata isaac onora ramaria aurea wikipedia
file zürich industriequartier prime...
my chikoo sapodilla noseberry mudapples...
industrial mobimo taweer
induttriel H. quactar for prinze ail intuan
chiiao
child
curbarr surtch
mopoasile
statue mentuhotep aa by khruner jog
カサンカー アリンク アンテイー2 AST4...
file athena promachos reggio calabria jpg
file birkenstock jpg
IX X XI XII
athena promschoe
II III IIII V VI
Iirrerstock Barannstock
VII
Figure A3: Example pseudo-annotation on WebLI [4]. Image- Insoe-associated text (from the HTML alt text tag) is shown above the images. If the text is not in English, an automatically generated translation used. N-grams are extracted from these texts to generate queries for the annotator model. Pseudo-annotations were filtered as for our main experiments: To be included, boxes must have score of at least 0.1, and images must have at least one box with a score above 0.3. All images from Wikimedia Commons.
21

---

Hh
فلسطين
فلسطين
Figure A4: Training inputs after pre-processing. Top: A 4 x 4 mosaic randomly resized and
padded images as used for self- self-training. Bottom: The same mosaic after dropping the 50% of patches with lowest pixel variance (image size: 1008 1008; patch size: 14 14). Most dropped patches belong to padding areas or uniform image backgrounds. All images from Wikimedia Commons.
22

---

OWL-ST L/14 self-traine on N N-grams, not fine-tuned (Table row 12)
Gamuun
mB
io GREA
Toncors
00
spole
Coka S
$2.99
SPOTE
BUCKET
BOOM
APPI
KoKukkat
Apple
Apple
apple
Eima
PSP
teara pople
Apple
OWL-ST+FT L/14 self tafintraned on N- grams and fine-tuned on LVIS (Table row 15)
Lightiii
AAGAGA
shopping Earth nach
de engge (hrutty
Th onange tragg
ordnge
omange (erutter ntatgee eragge thfunagee frait
Vat VAnna nal fry wrn in ani
prat Bmill
Sanket
Ket
arange Brunt
me
Rotat CALA
signtoord
2 0
GaanMt Smit
TOM
Hoote
spple
rate
cople
AD epple
anpiula
BIENN Busiio
epple
ample
pople
apple
Appue
spple
Appl
abako
spple
ppple
appe
pppie
Figure A5: Qualitative example for OWLv2 L/14 from the LVIS val set. For the visualization, all LVIS classes were used as prompts. LVIS classes are labeled in black. Top: OWL-ST self-traine trainer on N-grams, not fine-tuned (Table row 12). Bottom: OWL-ST+FT self-trained on N-grams and fine-tuned on LVIS (Table row 15). Boxes above score 0.08 (top) or 0.3 (bottom) are shown.
23

---

OWL-ST L/14 self-trained on N-grams, not fine-tuned (Table row 12)
OWL-ST+FT L/14 self-trained on N-grams and fine-tuned on L VIS (Table row 15)
Figure A6: Qualitative example for OWLv2 L/14 from the LVIS val set. For the visualization, all LVIS classes were used as prompts. LVIS classes are labeled in black. Top: OWL-ST self-trained on N-grams, not fine-tuned (Table I row 12). Bottom: OWL-ST+FT self-trained on N-grams and fine-tuned on LVIS (Table I row 15). Boxes above score 0.08 (top) or 0.3 (bottom) are shown.
24

---

OWL-ST 1/14 self trained on N- grams, not fine fune- (Table row 12)
Greca contieer
Trscal ooreseen
OWL-ST+FT 1/14 self trained on N N-grams and fine
fine-tuned tuned on LVIS (Table I row 15)
Figure A7: Qualitative example for OWLv2 L/14 from the LVIS val set. For the visualization, all LVIS classes were used as prompts. LVIS classes are labeled in black. Top: OWL-ST self-trained on N-grams, not fine-tuned (Table row 12). Bottom: OWL-ST+FT self-trained on N-grams and fine-tuned on LVIS (Table row 15). Boxes above score 0.08 (top) or 0.3 (bottom) are shown.
25


Reviews:
[{"review_id": "FO9ACVaZw3", "reviewer": "Reviewer_5UCC", "summary": "This work adopts a self-training strategy to generate web scale object annotations for open-vocabulary object detection. It uses an existing  open-vocabulary object detector (OWL-ViT) to annotate 10B image-text pairs, and uses the annotated dataset for self-training. A various architecture is also proposed to improve the training efficiency. Experimental results show a large performance gain on open-vocabulary detection.", "questions": "1. This paper is a coherent work of OWL-ViT. It's not straight forward for me to drop the DETR decoder in OWL, but rather predict a bounding box with each patch token. Such a dense prediction results in a significant number of background proposals (Instance selection in Sec. 3.2), which leads to inefficient detection.\n\n2. While N-grams tend to maintain a higher number of image-text alignments in comparison to rule-based methods (such as highest-scoring or threshold-based approaches), they also tend to generate more redundant alignments, particularly for obvious objects where the confidence scores are higher (e.g., both 'boy', 'a boy', 'a boy in red' might correspond to the same region). Does this bias the model towards learning obvious objects while potentially overlooking the hard examples?", "limitations": "The self-training strategy in the paper requires a large amount of computing resources and web-crawled data, which is not feasible for many colleges and research institutes. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A self-training strategy to annotate the bounding boxes for the large-scale image-text pairs. This provides an open-vocabulary detection dataset for various location-specific tasks, including detection, segmentation, visual grounding, etc. Despite self-training being widely explored in many previous works (discussed in Sec 2.3), this work provided a detailed ablation in the filtering strategy in Sec. 4.4.\n\n2. The performance is promising in rare classes, indicating the ability of zero-shot transfer with the self-training strategy.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Below, we address your questions and comments.\n\n## Response to Questions\n1. **_Is dropping the decoder inefficient?_** Our decoder-free architecture is actually more efficient than an equivalent encoder-decoder architecture like DETR. The number of encoder output tokens is the same in both cases. However, in a decoder model, every decoder layer cross-attends into the large number of encoder output tokens. Our encoder-only architecture only applies a lightweight objectness-prediction head to each encoder output token. Background tokens (= low objectness score) are then filtered out and do not need to be decoded into boxes (Instance Selection in Section 3.2). Therefore, the amount of processing that our model has to do per encoder output token is less than that of encoder-decoder models like DETR.\n\n2. **_Do redundant ngrams bias the model towards learning obvious objects?_** No, because obvious objects do not contribute more pseudo-annotations than hard objects. Consider your example of an image containing a boy: OWL-ViT would predict one box for the boy in the image, and then assign high probabilities for the ngrams \"boy\", \"a boy\", \"a boy in red\" etc. There would be a large number of high-confidence, redundant ngrams for that box. However, for self-training, we only choose the single highest ngram for that box. This way, a box with many redundant matching ngrams will not have a higher weight in training than a box that has only a single matching ngram. Additionally, we use non-maximum suppression to remove near-duplicate pseudo-annotations. Further, we do not use \"soft labels\" during self-training, i.e. we treat low-confidence and high-confidence pseudo-boxes the same (as long as they meet the inclusion threshold). We will clarify this in the paper.\n\n## Response to Limitations\n**_The self-training strategy in the paper requires a large amount of computing resources and web-crawled data, which is not feasible for many colleges and research institutes._** We would like to point out that our method provides substantial improvements already at amounts of training that are comparable to previous models and useful for resource-constrained researchers. In Figure 1 (right), the black cross indicates the amount of training used for the original OWL-ViT model. With similar training compute, and only about 250M additional web images, our L/14 model (orange line) improves over OWL-ViT by about 6 percentage points. Datasets with this amount of web data are publicly available (e.g. at https://laion.ai/). We therefore believe that this research is feasible with moderate resources and public data. Additionally, we will release self-trained model checkpoints."}, {"review_id": "ENX0npqVNL", "reviewer": "Reviewer_EGtY", "summary": "This paper presents OWLv2 model and OWL-ST self-training recipe for open-vocabulary object detection. The detection data is greatly enriched with the aid of self-training. Concretely, the authors use WebLI dataset as the source of weak supervision for self-training. The dataset consists of approximately 10B images and associated alt-text strings (noisy captions). Then OWL-ViT CLIP-L/14 is utilized to annotate all images with pseudo boxes. For self-training at scale, the authors present several techniques including Token dropping, Instance selection and Mosaics to make the training efficient. Experiments on LVIS show the effectiveness of the proposed approach.", "questions": "See weaknesses box.", "limitations": "The limitations are adequately discussed in Section 5.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper attemps to scale open-vocabulary object detection with self-training on large-scale weakly labeled dataset. This is a good engineering work built upon the existing detectors and datasets. This work verifies that large model + large data is a good option to enable zero-shot open-vocabulary detection.\n2. The experimental results look like promising. Result on LVIS dataset is good.\n", "weaknesses": "1. Authors should carefully polish their paper and fixtypos such as: In L12, L/14->VIT/14; L109, consist->consists.\n2. The authors are encouraged to discuss how to handle noise in self-training. What's effect without handling these noise.\n3. What's the inference process? The authors are encouraged to describe this in their next version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Below, we address your questions and comments.\n\n## Response to Weaknesses\n1. **_Typos:_** Thank you for identifying these typos, we will fix them and do an additional round of proof-reading.\n\n2. **_Discussion of noise in self-training:_** In the paper, we already discuss one way to handle noise in Section 4.4 (_Filtering of Pseudo-Annotations_). In this section, we filter pseudo-annotations by annotator confidence, which removes noisy annotations at the expense of biasing the remaining annotations to objects that the annotator is good at.\nWe find that strong filtering (e.g. confidence threshold 0.7) tends to be worse than weak filtering (e.g. threshold 0.3), which suggests that bias introduced by strong filtering is worse than the noise in the data. Based on the trends in Figure 3 (i.e. confidence threshold 0.1 is worse than 0.3), we believe that no filtering (i.e. not handling noise at all) also performs poorly, suggesting that some filtering is necessary but that the remaining noise is acceptable. We will expand this discussion in the paper. Please let us know if you have other analyses in mind.\n\n3. **_What's the inference process?_** The inference is the same as for the original OWL-ViT (see L187f: _\"Note that at inference, the model is identical to the original OWL-ViT.\"_). We will expand this sentence and provide some details rather than only referring to the original OWL-ViT paper."}, {"review_id": "r6abA5PRie", "reviewer": "Reviewer_tv8z", "summary": "The paper proposes a self-training recipe for open-vocabulary object detection that leverages weak supervision in the form of image-text pairs from the Web. The authors identify three key ingredients for optimizing the use of weak supervision for detection: choice of label space, filtering of pseudo-annotations, and training efficiency. They propose to use all possible N-grams of the image-associated text as detection prompts for that image and apply only weak confidence filtering to the resulting pseudo-labels. The self-training recipe is applied to the OWL-ViT detection architecture and is called OWL-ST. The authors introduce OWLv2, an optimized architecture with improved training efficiency. The proposed method surpasses prior state-of-the-art methods already at moderate amounts of self-training and achieves further large improvements when scaled to billions of examples. The authors also evaluate their models on a suite of \"in the wild\" datasets and study the trade-off between fine-tuned and open-vocabulary performance.", "questions": "Since the authors have mentioned SAM, I am curious about how it compares to SAM. Although one is a segmentation model and the other is a detection model, SAM can also be combined with Grounded, such as in https://github.com/IDEA-Research/Grounded-Segment-Anything, to achieve detection. I'm wondering how the authors view the comparison between their model and SAM in terms of certain task capabilities. This is not a necessary question to answer, I just want to have a friendly discussion with the authors.", "limitations": "The main issue is still the enormous amount of computation required. However, if it is open-sourced as a foundational model for public use, it would still make a significant contribution to the community.\n\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper is well-written, easy to understand, and very rigorous.\nThis paper thoroughly discusses the methods and challenges of extending OWL-VIT to the billion-level image scale.\nThis paper achieves surprising performance.", "weaknesses": "As the authors mentioned, the biggest issue with this paper is the significant amount of computation required, which makes it difficult for general research institutions to reproduce. Therefore, my question is whether the authors will make the research results available as a foundational model, similar to SAM, for public use. Additionally, can it create a wave of downstream tasks, like SAM did, by providing basic capabilities?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Below, we address your questions and comments.\n\n## Response to Weaknesses\nWe are happy to confirm that we will make checkpoints and code publicly available. We hope that it will be useful as a foundation for downstream tasks.\n\n## Response to Questions\nWe see our work as complementary to SAM and think that it has large potential when combined with SAM, similar to the Grounded Segment Anything that you mentioned. SAM performs highly accurate conditioned segmentation, but it requires some spatial conditioning information (e.g. a box or a point) to know what to segment. Conditioning on text is only discussed briefly in the SAM paper and is not quantitatively evaluated. In contrast, we focus on the semantic aspect of object recognition, i.e. matching text descriptions to objects in an image. A combination of the two models, where OWLv2 is used to obtain boxes matching text queries, and SAM is then used to predict masks for the detected boxes, would provide a strong open-vocabulary segmentation solution.\n\n## Response to Limitations\nOur fully trained models indeed required large amounts of compute. We will make these checkpoints publicly available to save other researchers the need for repeating the training. However, we would also like to point out that our method provides substantial improvements already at amounts of training that are comparable to previous models. In Figure 1 (right), the black cross indicates the amount of training used for the original OWL-ViT model. With similar training compute, and only about 250M additional web images, our L/14 model (orange line) improves over OWL-ViT by about 6 percentage points. We therefore make a useful contribution even at moderate amounts of compute."}, {"review_id": "4FfI5aEbvm", "reviewer": "Reviewer_F7mV", "summary": "The paper extends previous OWL-ViT detectors with self-training an 10B Web image-text data. The authors design different self-training strategies, such as token dropping, instance selection and Mosaics augmentation. It is an achievement to beat all previous methods on open-vocabulary benchmarks.", "questions": "- What's the meaning of the $AP^{mini}$ metric in Table 1? Does it mean the LVIS test set?", "limitations": "I like this paper which includes comprehensive experiments and excellent performance, so I think this paper deserves to be accepted. But to me, this paper is not pretty interesting since this paper is more like a technical report with lots of engineering efforts rather than a research paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors provide necessary implementation details about scaling up the open-vocabulary detector on massive web data with a self-training method.\n- The performance on the open-vocabulary LVIS benchmark is truly impressive.\n- The authors promise to open-source their code.", "weaknesses": "- GLIP2 performs better than OWL-ST on the ODinW dataset, can the authors explain the possible reasons?\n- The authors may consider adding the ablation studies about w or w/o the mosaic augmentation. The additional gain from the mosaic augmentation is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Below, we address your questions and comments.\n\n## Response to Weaknesses\n**_GLIP2 performs better than OWL-ST on the ODinW dataset, can the authors explain the possible reasons?_**\n\nGLIP2 uses much more human-annotated data than OWL-ST. Specifically, OWL-ST only uses Objects365 and Visual Genome (indirectly, through the annotator model). GLIP2 additionally trains on COCO, OpenImages, ImageNetBoxes, and GoldG (which includes 800'000 images with human grounding annotations from RefCOCO, RefCOCOg, and RefCOCO+). In addition, some of the tasks in ODinW are derived from OpenImages, so GLIP/GLIP2 is partially trained on ODinW evaluation data. We believe that these additional human annotations explain why GLIP reports higher numbers on ODinW, and will explore adding these to the OWL-ST training or fine-tuning data.\n\n**_The authors may consider adding the ablation studies about w or w/o the mosaic augmentation._**\n\nWe performed an ablation study on the mosaic augmentation and provide it in the rebuttal PDF. We will add it to the paper appendix. The results show that increasing the number of mosaic tiles improves overall LVIS AP (top left plot) at least up to $12 \\times 12$ (the maximum we tried). Improvements are primarily due to the \"rare\" classes and \"small\" objects. Note that the benefit of mosaics may be due to seeing smaller objects on average, or due to seeing more WebLI images per training step (a $12 \\times 12$ mosaic contains 144 WebLI images).\n\n## Response to Questions\n**_What's the meaning of the $AP^{mini}$ metric in Table 1? Does it mean the LVIS test set?_**\n\n$AP^{mini}$ refers to the LVIS \"minival\" set, which is a subset of the validation set and was introduced by MDETR (https://arxiv.org/pdf/2104.12763.pdf) and subsequently used by some object detection papers like GLIP/GLIPv2. We will clarify this in the table caption.\n\nWe report $AP^{mini}$ for comparability with the papers that only report that metric. Note that $AP^{mini}$ values are significantly different (usually higher) from the standard LVIS AP.\n\n## Response to Limitations\n**_...this paper is more like a technical report with lots of engineering efforts rather than a research paper._**\n\nGiven the focus on scaling in the title and text of the paper, we understand this impression. However, we want to emphasize that **we do not simply scale up existing methods. On the contrary, we provide new research results which show that prior approaches to self-training for object detection were not optimal. These new insights improve performance independently of massive scale.** Specifically, through systematic study, we find that weak filtering and simple label space construction outperform prior approaches like strong filtering (e.g. using only a single pseudo-box or query per image) or complex grammatical query parsing. Figures 2, 3, and 5 all provide new research results that are applicable to other tasks, already at moderate data and compute scales (also see response to Reviewer 5UCC). We therefore believe that our work is not only a technical scaling effort but provides a valuable update to the research on training localization models."}]

Author rebuttal:
We thank the reviewers for their useful feedback and comments.
The reviews were unanimously positive.
In our response, we address the remaining questions and concerns.
We provide an overview below and detailed responses to the individual reviewers.

In particular:

1. We confirm that we will release trained model checkpoints upon publication.
2. We now provide additional experiments to show the benefit of mosaics (see PDF; requested by Reviewer F7mV).
3. We clarify that we do not simply scale up existing methods, but provide new research results that challenge prior approaches to self-training and are useful independently of scale (see response to Reviewer F7mV).

We hope our work will be of value to the NeurIPS community both through the model release and through our research contributions.

Meta-review: The paper scales up open-vocabulary object detection with improved computation efficiency and impressive results. The rebuttal provides additional ablation studies and successfully resolves the minor technical concerns. Five reviewers unanimously give accept recommendations. Clear accept. Scaling to large data and OOD scenarios is the direction of deep learning, so it is beneficial to have this paper seen by larger audience at the conference as a spotlight.
Decision: Accept (spotlight)