Title: Scaling Open-Vocabulary Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: object detection

open-vocabulary object detection

vision transformers

vision-language models

scaling

self-training
Abstract: Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling. Code and checkpoints are available on GitHub.

Scaling Open-Vocabulary Object Detection

Abstract

Open-vocabulary object detection has benefited greatly from pretrained visionlanguage models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to imagelevel pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudoannotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales ( ≈ 10 M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With a ViTL/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations , from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling. Code and checkpoints are available on GitHub. 1

1 Introduction

Object detection is a core computer vision task with many real-world applications. Consequently, there is great interest in improving detection models, especially in the open-vocabulary domain. For image-level tasks, large improvements have been achieved through contrastive pretraining of visionlanguage models, which is massively scalable because it can use naturally abundant weak supervision in the form of image-text pairs from the Web [30, 12, 29]. Since no such natural supervision data exists for localization tasks, open-vocabulary detection models typically build on pretrained imagelevel encoders [9, 19, 26, 46, 22, 1, 39, 47]. However, due to the scarcity of detection data and the fragility of pretrained representations, detection-training stages of these models have typically had to be relatively brief, which limits final detection performance and scaling potential.

The scarcity of detection data can be addressed with self-training . In self-training, an existing detector is used to predict bounding boxes on unlabeled images to generate data for training better detectors [31, 48, 35]. By combining open-vocabulary detectors with Web image-text data, such pseudo-labeling can produce practically unlimited amounts of open-vocabulary detection training data that leverages the image-associated text for semantic supervision. While several works have applied various forms of self-training to open-vocabulary object detection [46, 1, 47, 39, 38], they have done so at relatively small scales, comparable to the size of human-annotated detection datasets and much smaller than the datasets used for image-level training.

OWL-ViT

detection loss

OWL-ViT

detection loss

Caption:

'Monarch on

a Zinnia'

N-grams:

Monarch

Monarch on

Monarch on a

Monarch on a Zinnia

…

Open-vocab

detector

(OWL-ViT L/14)

OWLv2

Pseudo-annotated

WebLI

1. Annotation

2. Self-training

3. Fine-tuning

(optional)

LVIS

base

OWLv2

Zinnia

Monarch on a Zinnia

10 7

10 8

10 9

Total examples seen

(including repetitions)

25

30

35

40

45

50

LVIS APrare

(%)

Unseen classes

Model

OWL-ST+FT G/14

OWL-ST+FT L/14

OWL-ST+FT B/16

OWL L/14

(annotator)

Best OWL

Best F-VLM

Best ViLD

Best 3Ways

Best OWL

Best F-VLM

Best ViLD

Best 3Ways

To scale detection self-training further, we take guidance from image-level methods, where the principle has been to leverage weak supervision in the largest possible amount [30, 12, 29, 42]. We identify three key ingredients for optimizing the use of weak supervision for detection: choice of label space, filtering of pseudo-annotations, and training efficiency. Prior methods have typically used human-curated label spaces or complex concept mining [47, 39, 38, 46] and strict filtering, keeping just the single largest [47] or highest-scoring [1] pseudo-box for each image. In contrast, we argue that we should 'let the data do the work' and therefore apply little processing and filtering. We propose to simply use all possible N-grams of the image-associated text as detection prompts for that image, and apply only weak confidence filtering to the resulting pseudo-labels.

We apply this self-training recipe to the OWL-ViT detection architecture [26] and call it OWL-ST. To increase the number of examples seen for a given amount compute, we also introduce OWLv2, an optimized architecture with improved training efficiency. Combining the OWL-ST recipe with the OWLv2 architecture surpasses prior state-of-the-art methods already at moderate amounts of self-training, comparable to training amounts of previous methods (Figure 1). Scaling self-training to billions of examples yields further large improvements. For example, our ViT-L/14-based model, trained on 2.3B image-text pairs and fine-tuned on LVISbase, achieves 44.6% zero-shot LVIS mAPrare, which is a 36% relative improvement over the prior state of the art (32.8% mAPrare for F-VLM R50x64 [19]). Our largest model, ViT-G/14, reaches 47.2% mAPrare.

We also evaluate our models on a suite of "in the wild" datasets [21] and study the trade-off between fine-tuned and open-vocabulary performance. We find that strong in- and out-of-distribution performance is possible with weight ensembling [37]. Finally, our analysis of the scaling behavior of OWL-ST suggests that self-training has further potential for leveraging abundantly available weak supervision for open-vocabulary object detection.

2 Related Work

2.1 Scaling Vision Models

Vision models have recently seen large advances in model and training scale, leading to improved performance on many image-level tasks. On the architecture side, Vision Transformers have been shown to scale more efficiently than prior architectures [17]. Task performance improves predictably as training data and compute are increased [42], with recent work showing continued improvements for models with up to 22 billion parameters [6]. We apply these findings to object detection.

On the data side, contrastive pretraining of vision-language models (VLMs) [30] has unlocked the use of abundantly available image-text pairs from the Web as weak supervision, with improved results if more data is used [12, 28]. VLMs, which embed images and text into a shared space, also enable open-vocabulary applications where prior models were limited to fixed label spaces. Here, we use pretrained CLIP [30] and SigLIP [43] encoders as backbones for our detector.

2.2 Open-Vocabulary Object Detection

Much recent work aims to transfer the open-vocabulary capabilities of VLMs to localization tasks such as object detection. A first wave of VLM-based object detection methods either distilled VLMpredictions for cropped image regions (e.g. ViLD [9]), or added detection heads directly to frozen ( F-VLM [19]) or fine-tuned ( OWL-VIT [26]) VLM encoders. A challenge identified by these works is to protect the VLM from forgetting its open-vocabulary knowledge while training the detection heads on the relatively little available detection data.

2.3 Scaling Open-Vocabulary Detection with Weak Supervision

Given that earlier methods identified detection data as a limiting factor in open-vocabulary detection performance, more recent works focus on using weak supervision directly for detection training, rather than just during VLM pretraining. There are two main approaches:

Some methods use self-training , in which an existing detector is used to predict pseudo-boxes for images where image-level labels or captions, but no human box annotations, are available. Better detectors can then be trained on the pseudo-annotations. For example, RegionCLIP [46] generates pseudo-boxes using nouns parsed from image captions and uses those boxes for localization pretraining. Detic [47] predicts class-agnostic pseudo-boxes on images for which classification labels are available and associates the largest predicted box with the image label. Similar to our approach, 3Ways [1] uses an existing open-vocabulary detector to predict pseudo-boxes on captioned images, but uses the whole caption as a prompt, instead of dividing it into multiple prompts as we do.

Other methods propose grounding losses that directly train a detector on weak supervision such as image-level labels or captions. These methods pretrain models to align class-agnostic pseudo-boxes with words from image-associated text and rely on human-generated detection data for fine-tuning. Major examples of this approach are GLIPv1/v2 and [22, 45] and DetCLIPv1/v2 [39, 38].

In principle, these approaches unlock Web-scale training for detection, but prior methods rarely go much beyond 10M examples and instead focus on the model architecture and training loss. Here, we keep architecture and loss simple, and focus on scaling up the training data, since this was successful for image-level models. A similar approach was recently applied with good results to class-agnostic segmentation in the Segment Anything work [16]. Together with our results on text-conditioned localization, this suggests that scaling up self-training is a powerful and general method for improving performance on fine-grained vision tasks.

3 Method

We propose a simple self-training approach with three steps: (1) Use an existing open-vocabulary detector to predict bounding boxes for a large Web image-text dataset. (2) Self-train a new detector on the pseudo-annotations. (3) Optionally, fine-tune the self-trained model briefly on human-annotated detection data (Figure 1, left). Our goal is to optimize the key components of this approachlabel space, annotation filtering, and training efficiency-such that it provides strong and scalable open-vocabulary performance with few human annotations.

3.1 Generating Web-Scale Open-Vocabulary Object Annotations

We use the WebLI dataset [4] as the source of weak supervision for self-training. WebLI is a large dataset of images and texts available on the public Web. The dataset consists of approximately 10B images and associated alt-text strings, which can be thought of as noisy image captions. For images whose alt-text is not in English, we use an automatically generated English translation [4].

We use OWL-ViT CLIP-L/14 [26] to annotate all 10B WebLI images with bounding box pseudoannotations. OWL-ViT is an open-vocabulary object detector. Given an image, the model first detects objects in the image in a class-agnostic way. Then, given a list of free-text queries, the model produces scores indicating the likelihood that each detected object is associated with each text query.

A crucial design choice for open-vocabulary pseudo-labeling is the annotation label space. Methods in the literature vary widely but typically fall somewhere between two extremes: (1) use a fixed, human-curated label space for all images (e.g. [47]), or (2) machine-generate per-image queries from image-associated text (e.g. [1]). We implement both and compare their performance in Section 4.3.

Human-curated label space. We performed one pseudo-annotation run by combining the label sets from the LVIS [10], Objects365 [33], OpenImagesV4 [20], and Visual Genome [18] datasets and removing duplicates and plural forms. In total, this label space contains 2520 common object categories, e.g. "phone" , "goatee" , "teakettle" , "park" , "suit (clothing)" . See Appendix A.2 for code to generate the full list. Models trained on this label space may not be considered fully open-vocabulary for evaluation datasets whose classes were included in the pseudo-annotation label space (e.g. LVIS), since the evaluation vocabulary is known at training time in this case. However, LVISrare classes are still unseen for all of our models, in the sense that neither the annotator nor the self-trained models have ever seen human box annotations for LVISrare classes.

Machine-generated label space. In a second pseudo-annotation run, we automatically generated queries from the image-associated text. Prior work using image captions as weak supervision for detection often used grammatical parsing to extract noun phrases or concepts [46, 39, 38]. These approaches may add biases that reduce the diversity of extracted queries. To keep such biases to a minimum, we use no grammatical parsing and simply extract all word N-grams up to length 10 from the text associated with a given image and use them as queries for that image. We apply minimal filtering, only removing generic terms like image or png , and queries consisting entirely of stop-words (details in Appendix A.3). Note that, since OWL-ViT uses late image-text fusion, the quality of box localization (as opposed to classification) is not affected by the chosen label space.

Regardless of label space, we ensemble predictions over seven prompt templates such as "a photo of a {}" as described in [26]. For each predicted box, we keep the query with the highest score as its pseudo-label. For each image, we keep all boxes above a score threshold. We study the choice of threshold in Section 4.4. The pseudo-annotations are used as hard labels for self-training.

3.2 Self-training at Scale

We now describe how we use the pseudo-annotations to train better detectors. We use a variant of the OWL-ViT architecture [26] as described below. The image and text encoders are initialized from contrastively trained image-text models (CLIP, unless noted otherwise); the detection heads are randomly initialized. All models are first trained exclusively on pseudo-annotations ('self-training'). In an optional separate step, models are fine-tuned briefly on human-annotated detection data.

Self-training proceeds similarly to detection training in [26]. In particular, we use the same losses and also augment queries with 'pseudo-negatives' that are randomly sampled from the queries of other images, similar to batch negatives in [1]. Due to the size of our dataset, in contrast to [26], we use no random prompt templates and fewer image augmentations (details in Appendix A.5).

Prior work on image-level tasks shows that pretraining improves performance on downstream tasks well beyond 1 billion examples seen [44, 12, 28, 42], across model sizes. We hypothesize that similar scaling applies to detection self-training. We therefore optimize training efficiency to maximize the number of images seen for a given amount of training compute as follows.

Token dropping. Vision Transformers represent images as an unordered sequence of tokens. Tokens can therefore be reorganized or dropped without changing the model parameters. Various forms of token dropping or pooling have been proposed to improve efficiency [24, 32, 41, 25, 2]. Here, we drop tokens simply based on the pixel variance of the corresponding image patch. Both natural and Web images contain low-variance areas devoid of useful information, e.g. sky, single-color backgrounds, or padding. We find that the lower half of image patches by mean pixel variance can be dropped without loss in detection performance (Appendix A.6). We therefore drop 50% of patches in all of our experiments during training. No patches are dropped during inference.

Instance selection. OWL-ViT is an encoder-only architecture and predicts one bounding box per encoder token. This is inefficient, since there are typically many more encoder tokens than objects (e.g. 5184 tokens for resolution 1008 × 1008 and patch size 14 × 14 ). Most output tokens therefore do not represent objects. We introduce an objectness head which predicts the likelihood that an output token actually represents an object, and compute boxes, class scores, and losses only for the top k tokens by objectness, similar to Efficient DETR [40]. The objectness head receives an encoder token as input and computes a scalar objectness score. The objectness score predicts the future classification score of a token and is supervised by the actual classification score of those tokens that end up being selected and passed on to the classification head. We select approximately 10% of instances by top objectness during training in all of our experiments. During inference, all instances are used.

Mosaics. During self-training, we combine raw images into grids of up to 6 × 6 to produce a single training example (i.e. a more extreme version of the mosaics in [26]). This has two main motivations: (1) Using mosaics increases the number of raw images seen for a given fixed model input resolution. An alternative is to train using variable image sizes [38], but this would require resizing image position embeddings for each input size. (2) The average resolution and complexity of Web images is lower than images in detection benchmarks and applications. Mosaics reduce the average object size and improve small-object performance, similar to large scale-jittering [8], but with less padding. For all self-training experiments, we use 1 × 1 , 2 × 2 , 3 × 3 , 4 × 4 , and 6 × 6 grids in equal proportions, resulting in an average of 13.2 raw component images per training example.

To further improve training efficiency, we also adopt previously proposed practices for large-scale Transformer training [42] (details in Appendix A.7). Together, our improvements reduce training FLOPS by approximately 50% compared to the original OWL-ViT [26] and increase training throughput by 2 × (e.g. for L/14 at 840 × 840 resolution measured on TPUv3: GFLOPs/example 11'945.4 vs. 5357.9; examples/s/core 1.0 vs. 2.2). We refer to the improved model as OWLv2.

At inference, no token dropping or instance selection is performed. Inference is therefore identical to the original OWL-ViT, i.e. each image encoder token is decoded into a bounding box and a list of per-query classification scores.

10 7

10 8

10 9

Total examples seen

(including repetitions)

25

30

35

40

45

50

LVIS APfrequent

(%)

Fine-tuned classes

10 7

10 8

10 9

Total examples seen

(including repetitions)

15

20

25

30

35

40

45

LVIS APrare

(%)

Unseen classes

10 7

10 8

10 9

Total examples seen

(including repetitions)

30

35

40

45

50

ODinW13 mean AP (%)

"In the Wild" datasets

Pseudo-label space

Curated vocabulary

N-grams

N-grams+curated

3.3 Fine-tuning

Self-training on pseudo-annotations alone already yields strong performance (Section 4.2). However, fine-tuning briefly on human annotations can provide significant further benefits. For fine-tuning, we start with the learning rate and optimizer state of the self-trained checkpoint and then continue training on the target dataset while linearly cooling down the learning rate to zero. Fine-tuning of open-vocabulary models involves a trade-off between improving the performance on the fine-tuned classes and losing open-vocabulary performance [30, 29, 37]. We study this trade-off in Section 4.6.

4 Experiments

4.1 Experimental Setup

Models. We use the publicly available OWL-ViT CLIP L/14 model to generate detection pseudoannotations for the WebLI dataset (10 billion image-text pairs [4]). For all self-training experiments, we use OWL-ViT models modified as described in Section 3.2. Backbones are initialized with the publicly available CLIP [30] checkpoints (B/16 and L/14) or a SigLIP [43] checkpoint (G/14).

Training. Models are first self-trained on the pseudo-annotations for varying durations as indicated. If indicated, after self-training, models are fine-tuned on LVISbase, i.e. the LVIS dataset [10] with all annotations for 'rare' categories removed. Therefore, neither the annotator nor any of our models have seen human-generated annotations for LVISrare classes. Fine-tuning uses mosaics up to 3 × 3 and is always done until the model has seen 256'000 mosaics (1.1M individual images, roughly equivalent to 100 LVIS epochs). The image size is 960 × 960 for /16 models and 1008 × 1008 for /14 models. See Appendix A.8 for a complete list of hyperparameters.

Evaluation. We use mean average precision (mAP) on LVIS [10] as our main detection metric, where mAPrare indicates open-vocabulary performance on unseen classes. To measure generalization on diverse real-world tasks, we evaluate zero-shot performance on the 'Object Detection in the Wild' (ODinW) benchmark [21]. ODinW is a suite of datasets covering a wide range of domains. We report the average mAP on the subset of 13 ODinW datasets introduced in [22] and provide performance on individual datasets in Appendix A.9.2. To avoid leakage of evaluation data into the training set, WebLI was filtered to remove images similar to those in the train, validation, and test splits of 68 common computer vision datasets, including COCO/LVIS, Objects365, and Visual Genome, but not the ODinW datasets (see [4] for details).

10 7

10 8

Total examples seen

(including repetitions)

25

30

35

40

45

LVIS APfrequent

(%)

Fine-tuned classes

10 7

10 8

Total examples seen

(including repetitions)

15

20

25

30

35

LVIS APrare

(%)

Unseen classes

10 7

10 8

Total examples seen

(including repetitions)

30

35

40

45

ODinW13 mean AP (%)

"In the Wild" datasets

Confidence

threshold

0.1

0.3

0.5

0.7

4.2 Main Result

We compare our best models to the literature in Table 1. We broadly include state-of-the-art openvocabulary detectors in the comparison. Our self-training approach, using only machine-generated pseudo-annotation queries, improves over previous methods even without fine-tuning (Table 1, OWLST, rows 11-13). Our OWL-ST B/16 model (row 11) achieves 29.6% LVIS mAPrare, 9 points more than the equivalent OWL-ViT model (row 2). Our largest model, G/14 (row 13), reaches 37.5% mAPrare, 4.7 points better than the next-best model from the literature (F-VLM R50x64, row 8). Interestingly, after self-training, our models perform better on LVIS mAPrare than mAPall (which includes frequent and common classes). We speculate that this may be because weak Web-data supervision may be better for specific terms than general terms: Image/text pairs involving unusual objects (such as LVISrare categories) may be more likely to be specifically about these objects, whereas common terms like 'person' or 'car' may occur often without being related to the image.

Fine-tuning on LVISbase provides additional significant improvements, even on mAPrare (OWLST+FT, rows 14-16). Our best model, which has only seen machine-generated queries during self-training, reaches 47.2% LVIS mAPrare after fine-tuning, a 14.4-point improvement over the next best model (F-VLM R50x64, row 8).

Including a human-curated list of common object classes as pseudo-annotation queries can further improve the results on LVIS (rows 20-21), but this approach is not fully open-vocabulary since the model sees a curated label space, including the LVIS classes, at training time. While the benefit of the curated label space is significant for our smallest model, is is minor on mAPrare for the larger L/14 model (compare rows 15 and 21).

To measure more general open-world performance, Table 1 also includes zero-shot results on ODinW13 [21], a suite of 'in the wild' datasets. Performance on ODinW is best right after selftraining and is reduced by fine-tuning on LVISbase. We discuss this further in Section 4.6. We also fine-tuned on COCO, where our B/16 and L/14 models reach 54.3% and 56.0% COCO mAP, respectively. OWLv2 therefore matches the performance of ViTDet with a Cascade Mask-RCNN head [23], despite using a simpler head architecture. Further results and examples in Appendix A.9.

4.3 Pseudo-Annotation Label Space

Figure 2 takes a closer look at the impact of the pseudo-annotation label space on performance after fine-tuning. Performance on fine-tuned classes (mAPfrequent; left plot) is highest if the pseudoannotation label space included these classes (blue circles). Therefore, if the target label space is known ahead of time, pseudo-labeling on that space leads to the best results.

However, performance on unseen classes (mAPrare) and 'In the Wild' datasets is much better if the pseudo-labeling included diverse queries that were machine-generated from the image-associated text (orange squares and green diamonds). A mixture of human and machine-generated label spaces

10 1

10 2

10 3

Detection training compute

(exaFLOPs)

30

35

40

45

50

LVIS APfrequent

(%)

Fine-tuned classes

10 1

10 2

10 3

Detection training compute

(exaFLOPs)

20

25

30

35

40

45

LVIS APrare

(%)

Unseen classes

10 1

10 2

10 3

Detection training compute

(exaFLOPs)

30

35

40

45

50

ODinW13 mean AP (%)

"In the Wild" datasets

Model

OWL-ST+FT B/16

OWL-ST+FT L/14

OWL L/14 (annotator)

performs well in all settings, but does not significantly outperform the purely machine-generated label space on the 'In the Wild' datasets. These results suggest that a human-curated label space can help if the target label space is known, but that strong in-the-wild generalization is driven by the weakly supervised machine-generated label space. Our results also show that a simple N-grams approach is sufficient to leverage the weak supervision and outperforms more complex methods (Table 1).

4.4 Filtering of Pseudo-Annotations

Besides the label space, a second important decision in self-training is the filtering of pseudoannotations. We filter based on the detection confidence score of the annotator and vary the score threshold in Figure 3. For confidence-based filtering, a bias-variance trade-off exists between including only high-confidence pseudo-annotations but inheriting the annotator's biases, or lowering the bias but increasing the noise by including lower-confidence pseudo-annotations. Many prior works err on the side of high bias and low variance, applying high confidence thresholds [35] or including only the single highest-confidence detection for an image [47, 1]. In our setting, we find that including all pseudo-annotations that pass a moderate threshold of 0.3 works well, while strict thresholds lead to poor results (Figure 3). As training continues for longer than what was possible for Figure 3, we suspect that lower thresholds may scale better. Therefore, for our main results, we chose to include all annotations above 0.1, but only kept images with at least one annotation above 0.3.

4.5 Scaling

The use of abundant Web image-text data with little filtering means that our self-training dataset is large (approximately 2B images). We can therefore study detection training scaling in the same regime as prior work on classification (Figure 4; models see each image at most once for these experiments). We make several noteworthy observations:

Self-training is beneficial already at moderate compute budgets, less than that of the annotator.

Models show similar scaling behavior for detection as for classification [42]: Both overall performance and the size of the Pareto-optimal model increase with compute/data size.

As we move further out of distribution, the amount of compute at which L/14 overtakes B/16 increases. In other words, for in-the-wild performance, at most compute budgets, it may be better to train a smaller model for longer than a larger model for shorter.

These results suggests that self-training on Web data is further scalable as an approach for improving open-vocabulary localization models without the need for further human annotations. The large datasets also makes it possible to scale model size. We trained a G/14 model, which has 5 . 2 × the number of parameters and 4 . 3 × the inference FLOPs of our L/14 model. To our knowledge, this is the largest open-vocabulary detection model to date. Since the G/14 model uses a different backbone than our other models (SigLIP [43] instead of CLIP [30]), we do not include it in Figure 4, but show in Table 1 that it is currently the best-performing model on zero-shot LVIS, with 47.2% mAPrare.

44

46

48

50

52

54

"In the Wild" performance

(ODinW13 mean AP (%))

25

30

35

40

45

50

55

LVIS APfrequent

(%)

100

1k

2k

5k

20k

.1

.2

.3

.4

.5

.6

.7

.8

.9

Longer self-training →

Ensembles

44

46

48

50

52

54

"In the Wild" performance

(ODinW13 mean AP (%))

25

30

35

40

45

LVIS APrare

(%)

100

1k

2k

20k

.1

.2

.3

.4

.5

.7 .6

.8

.9

Self-training only

Fine-tuned on LVIS base

Weight ensemble

OWL L/14 (annotator)

4.6 Effect of Fine-Tuning on Open-Vocabulary Performance

For contrastively trained image-text models, fine-tuning improves performance on the target distribution but reduces the (originally very high) robustness to distribution shift [30, 28, 37]. We observe the same effect for detection, using ODinW13 AP as a proxy for out-of-distribution performance: Compared to the performance after self-training (red dots in Figure 5), fine-tuning on LVISbase improves performance on the fine-tuned classes (LVIS mAPfrequent), but OOD performance (ODinW13 AP) is simultaneously reduced in proportion to the amount of fine-tuning (light blue line in Figure 5).

A simple approach to improve on this trade-off is to create an ensemble of the model before and after fine-tuning by averaging the model weights [37]. This approach comes at no additional training cost and improves the Pareto-frontier for all ensemble mixing ratios (Figure 5, purple line). We also tried co-training on WebLI and LVIS but found it to perform worse than weight ensembling.

Notably, performance on LVISrare behaves similarly to LVISfrequent and improves during fine-tuning, even though no LVISrare classes are seen (Figure 5, right). This may be because LVISrare classes are semantically and visually close to LVISfrequent classes. For example, seeing many annotations for "bird" may improve performance on rare classes such as "heron" , "mallard" , or "puffin" . LVIS mAPrare therefore only measures a narrow concept of open-vocabulary performance, and does not reveal the fact that fine-tuning significantly reduces generalization to broader distribution shifts. Benchmarks such as ODinW therefore provide significant additional insight.

5 Limitations

The main limitation of our method is the amount of compute and data needed for self-training. As we show in Section 4.5, performance improves consistently with training compute and data. This means that further improvements are possible, but also that these will come at increasingly large costs. In fact, cost likely increases faster than resources can realistically be grown in practice. New approaches will therefore be eventually necessary for further improvements.

A second important limitation of our method, similar to other open-vocabulary models [30, 28, 37], is the trade-off between fine-tuned and open-vocabulary performance addressed in Section 4.6. For out-of-distribution queries, predictions of fine-tuned models may be poorly calibrated and may depend on the precise wording of the query. These issues can be mitigated with weight ensembling [37], but more research is needed to fully understand the open-vocabulary robustness of these models.

6 Conclusion

In the past, open-vocabulary detection performance has been limited by the availability of humanannotated detection training data. Here, we show that self-training can be scaled up to overcome the dependency on human annotations. Our OWL-ST recipe delivers large improvements in detection performance using weak supervision from abundant Web data, similar to what has been seen for image classification and language modelling.

Acknowledgments and Disclosure of Funding

We would like to thank Xiao Wang for help with the WebLI dataset, Xiaohua Zhai and Lucas Beyer for providing the SigLIP model, and Rich Munoz and Alexey Dosovitskiy for insightful comments.

References

Arandjelovi´ c, R., Andonian, A., Mensch, A., Hénaff, O.J., Alayrac, J.B., Zisserman, A.: Three ways to improve feature alignment for open vocabulary detection. arXiv preprint arXiv:2303.13518 (2023)

Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 (2022)

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M.J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., Zhang, Q.: JAX: composable transformations of Python+NumPy programs (2018), http://github.com/google/jax

Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A.V., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B.K., Ruiz, C.R., Steiner, A.P., Angelova, A., Zhai, X., Houlsby, N., Soricut, R.: PaLI: A jointly-scaled multilingual language-image model. ICLR (2023)

Dave, A., Dollár, P., Ramanan, D., Kirillov, A., Girshick, R.: Evaluating large-vocabulary object detectors: The devil is in the details. arXiv preprint arXiv:2102.01066 (2021)

Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Riquelme, C., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., van Steenkiste, S., Elsayed, G.F., Mahendran, A., Yu, F., Oliver, A., Huot, F., Bastings, J., Collier, M.P., Gritsenko, A., Birodkar, V ., Vasconcelos, C., Tay, Y ., Mensink, T., Kolesnikov, A., Paveti´ c, F., Tran, D., Kipf, T., Luˇ ci´ c, M., Zhai, X., Keysers, D., Harmsen, J., Houlsby, N.: Scaling vision transformers to 22 billion parameters. ICML (2023)

Dehghani, M., Gritsenko, A.A., Arnab, A., Minderer, M., Tay, Y.: SCENIC: A JAX library for computer vision research and beyond. arXiv preprint arXiv:2110.11403 (2021)

Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.Y ., Cubuk, E.D., Le, Q.V ., Zoph, B.: Simple copy-paste is a strong data augmentation method for instance segmentation. CVPR (2021)

Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 (2021)

Gupta, A., Dollar, P., Girshick, R.: LVIS: A dataset for large vocabulary instance segmentation. CVPR (2019)

Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., van Zee, M.: Flax: A neural network library and ecosystem for JAX (2020), http://github.com/google/flax

Jia, C., Yang, Y., Xia, Y ., Chen, Y .T., Parekh, Z., Pham, H., Le, Q., Sung, Y .H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. ICML (2021)

Jouppi, N.P., Yoon, D.H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., Patterson, D.: A domainspecific supercomputer for training deep neural networks. Communications of the ACM 63 (7), 67-78 (2020)

Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: MDETR - modulated detection for end-to-end multi-modal understanding. ICCV (2021)

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)

Kolesnikov, A., Dosovitskiy, A., Weissenborn, D., Heigold, G., Uszkoreit, J., Beyer, L., Minderer, M., Dehghani, M., Houlsby, N., Gelly, S., Unterthiner, T., Zhai, X.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR (2021)

Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y ., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123 (1), 32-73 (2017)

Kuo, W., Cui, Y., Gu, X., Piergiovanni, A., Angelova, A.: Open-vocabulary object detection upon frozen vision and language models. ICLR (2023)

Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., Ferrari, V .: The Open Images Dataset V4. International Journal of Computer Vision 128 (7), 1956-1981 (Mar 2020)

Li*, C., Liu*, H., Li, L.H., Zhang, P., Aneja, J., Yang, J., Jin, P., Lee, Y .J., Hu, H., Liu, Z., et al.: Elevater: A benchmark and toolkit for evaluating language-augmented visual models. NeurIPS (2022)

Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. arXiv preprint arXiv:2112.03857 (2021)

Li, Y., Mao, H., Girshick, R., He, K.: Exploring plain vision transformer backbones for object detection. arXiv preprint arXiv:2203.16527 (2022)

Liang, Y., Ge, C., Tong, Z., Song, Y., Wang, J., Xie, P.: Not all patches are what you need: Expediting vision transformers via token reorganizations. ICLR (2022)

Marin, D., Chang, J.H.R., Ranjan, A., Prabhu, A., Rastegari, M., Tuzel, O.: Token pooling in vision transformers for image classification. CVPR (2023)

Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., Houlsby, N.: Simple open-vocabulary object detection with vision transformers. ECCV (2022)

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., Gebru, T.: Model cards for model reporting. In: Proceedings of the conference on fairness, accountability, and transparency. pp. 220-229 (2019)

Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A.W., Yu, J., Chen, Y.T., Luong, M.T., Wu, Y., Tan, M., Le, Q.V.: Combined scaling for zero-shot transfer learning (2021)

Pham, H., Dai, Z., Ghiasi, G., Liu, H., Yu, A.W., Luong, M.T., Tan, M., Le, Q.V.: Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050 (2021)

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. ICML (2021)

Ramanathan, V., Wang, R., Mahajan, D.: Dlwl: Improving detection for lowshot classes with weakly labelled data. CVPR (2020)

Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.J.: Dynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS (2021)

Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.: Objects365: A Large-Scale, High-Quality Dataset for Object Detection. ICCV (2019)

Shazeer, N., Stern, M.: Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235 (2018)

Sohn, K., Zhang, Z., Li, C.L., Zhang, H., Lee, C.Y., Pfister, T.: A simple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.04757 (2020)

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. NeurIPS (2017)

Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Gontijo-Lopes, R., Hajishirzi, H., Farhadi, A., Namkoong, H., Schmidt, L.: Robust fine-tuning of zero-shot models. CVPR (2022)

Yao, L., Han, J., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, H.: Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. arXiv preprint arXiv:2304.04514 (2023)

Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., Xu, H.: DetCLIP: Dictionaryenriched visual-concept paralleled pre-training for open-world detection. NeurIPS (2022)

Yao, Z., Ai, J., Li, B., Zhang, C.: Efficient detr: Improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318 (2021)

Yin, H., Vahdat, A., Alvarez, J., Mallya, A., Kautz, J., Molchanov, P.: Adavit: Adaptive tokens for efficient vision transformer. CVPR (2022)

Zhai, X., Kolesnikov, A., Houlsby, N., Beyer, L.: Scaling vision transformers. CVPR (2022)

Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343 (2023)

Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: LiT: Zero-shot transfer with locked-image text tuning. arXiv preprint arXiv:2111.07991 (2021)

Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li, L.H., Dai, X., Wang, L., Yuan, L., Hwang, J.N., Gao, J.: Glipv2: Unifying localization and vision-language understanding. NeurIPS (2022)

Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al.: RegionCLIP: Region-based language-image pretraining. arXiv preprint arXiv:2112.09106 (2021)

Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., Misra, I.: Detecting twenty-thousand classes using image-level supervision. arXiv preprint arXiv:2201.02605 (2021)

Zoph, B., Ghiasi, G., Lin, T.Y., Cui, Y., Liu, H., Cubuk, E.D., Le, Q.V.: Rethinking pre-training and self-training. NeurIPS (2020)

A Appendix

The Appendix provides a Model Card [27] for OWLv2 as well as additional methodological details, hyperparameters, and results. At the end of the Appendix, we provide qualitative examples of the self-training data and model predictions.

The Appendix is structured as follows:

A.1 Model Card

A.2 Human-Curated Label Space

The human-curated label space was obtained by merging common dataset class lists with the Python code below.

A.3 Machine-Generated Label Space

The machine-generated label space was obtained from the image-associated text, for each image separately, using the Python code below. Figure A3 shows example pseudo-annotations using the N-gram label space.

A.4 Combined Label Space

When merging pseudo-annotations obtained with human-curated and machine-generated queries, it is important to consider that human-curated queries tend to be closer to the training distribution of the annotator and therefore tend to have higher scores than pseudo-annotations based on machinegenerated queries. Simply merging annotations from the two label spaces and filtering them with the same confidence threshold would therefore retain primarily annotations based on human-curated queries. To achieve a more even balance when using the combined label space ('N-grm+curated' in Table 1), we therefore re-scaled scores of pseudo-annotations obtained with the human-curated queries by a factor of 0.3 before applying the same confidence threshold to all (human-curated and machine-generated) annotations.

A.5 Augmentations for Self-Training

Since Web-scale image-text data differs in important aspects from human-curated detection datasets, we depart from the augmentation strategy of [26] in several ways. As described in Section 3.2, since Web images tend to be smaller and show fewer objects than e.g. LVIS images, we use stronger image mosaics with up do 6 × 6 tiles (Figure A1). For the same reason, we additionally randomly resize each raw image such that its width is between 0 . 5 × and 1 . 0 × the width of the full mosaic tile, padding on the bottom and right to preserve the aspect ratio (Figure A4).

On the other hand, given the large size of our dataset, some other augmentations can be avoided: We do not use left/right flipping or random cropping during self-training. We also do not add random

3×3

6×6

8×8

12×12

29

30

LVIS APall

(%)

All

3×3

6×6

8×8

12×12

32

LVIS APfrequent

(%)

Frequent

3×3

6×6

8×8

12×12

25

26

27

LVIS APrare

(%)

Rare

3×3

6×6

8×8

12×12

48

49

LVIS APlarge

(%)

Large

3×3

6×6

8×8

12×12

33

34

LVIS APmedium (%)

Medium

3×3

6×6

8×8

12×12

15

16

LVIS APsmall

(%)

Small

prompt templates to the pseudo-labels during self-training. During fine-tuning, we use the same augmentations as [26].

A.6 Token Dropping

To improve training efficiency, we drop image patches based on their pixel variance (Section 3.2). Table A2 shows how the performance of a standard OWL-ViT model varies for different amounts of token dropping. Dropping up to 50% of tokens is within one standard deviation of the full performance. We therefore drop 50% of tokens during all of our experiments.

To inject some stochasticity to the patch selection, we add a small amount of noise to the image before computing patch variance (uniformly distributed between 0.0 and 0.01 for images in the range [0 . 0 , 1 . 0] ). Figure A4 shows an example training image before and after token dropping.

A.7 Further Efficiency Improvements

To further improve training efficiency beyond the methods described in Section 3.2, we also adopt previously proposed methods for large-scale Transformer training: To save memory, we use a variant [42] of the Adafactor optimizer [34] instead of Adam [15]. To avoid having to choose and optimize the total training duration ahead of time, we use the open-ended inverse square-root schedule [36, 42] with a fixed time-scale of 10'000 steps for all experiments and linearly 'cool down' checkpoints along the way for evaluation (see Section 3.3).

A.8 Model Hyperparameters

We use the following hyperparameters for all of our models. Hyperparameters that vary between models are listed in Table A3.

Optimizer: Adafactor variant as in [42]

Learning rate schedule: Inverse square-root [36] with timescale 10'000 steps

Learning rate for the text encoder: 2 × 10 -6

Token dropping rate during training: 0.5

Pseudo-annotation confidence score threshold: 0.3 (except for Figure 3)

Augmentations: See Appendix A.5

All remaining hyperparameters are as in [26].

Hyperparameter selection. Most hyperparameters were either taken directly from [26] or technically constrained, e.g. we chose the largest batch size that fit into the memory of the available accelerators. Where hyperparameters were tuned, we ran short B/16-scale trial experiments and selected the parameters with the highest LVIS mAPrare for our main runs.

SigLIP G/14. For the G/14 model, we started self-training with a learning rate of 5 × 10 -5 , a droplayer rate of .1/.0, and no dropout. We found that the model overfit during fine-tuning with these settings, and switched to a learning rate of 2 × 10 -5 , a droplayer rate of .2/.4, and a dropout rate of .0/.1 after 740'000 self-training steps. To save resources, we did not start training from the beginning. With the new settings, we observed no overfitting during fine-tuning, but it is possible that these settings are still not optimal.

A.9 Additional Results

A.9.1 Fixed Average Precision

In the standard Average Precision metric (AP old ), performance on one class depends on the performance on other classes. This dependence makes the metric 'gameable' by re-scaling the scores of certain classes [5]. To avoid this issue, some prior work reports a 'fixed' version of AP proposed in [5]. In Table 1, we report AP old for our models. For models from the literature, we report whichever AP version is available. Since AP fixed tends to produce higher values than AP old , Table 1 tends to underestimate the advantage of our method over prior work using AP fixed . We provide AP fixed for all of our models in Table A4. As proposed in [5], we implement AP fixed by evaluating AP on the top 10'000 predictions per class over the entire validation set. This ensures that classes do not compete with each other for inclusion in the evaluated predictions.

A.9.2 Per-Dataset ODinW Results

Table A5 shows un-aggregated results on all 35 ODinW datasets for our main models. In addition, in the last row, we provide results for a weight-space ensemble of a self-trained and fine-tuned OWLv2 L/14 model (the same model is shown in Figure A2).

A.9.3 Fine-Tuning Robustness Trade-Off for OWLv2 L/14

In Figure A2, we provide the same analysis of the robustness trade-off after fine-tuning for an L/14 model that we provided for a B/16 model in Figure 5.

A.10 Qualitative Examples

In Figures A5 to A7, we provide qualitative examples of detection predictions from OWLv2 L/14 models. In each figure, the top image shows predictions obtained directly after self-training, and

the bottom image shows predictions after fine-tuning on LVISbase. Example images are from the LVIS validation set and the model was queried with all LVIS classes. All predictions meeting the confidence threshold specified in the caption are shown.

44

46

48

50

52

54

56

58

"In the Wild" performance

(ODinW13 mean AP (%))

30

35

40

45

50

55

LVIS APfrequent

(%)

100

1k

2k

5k

20k

.1

.2

.3

.4

.5

.6

.7

.9 .8

44

46

48

50

52

54

56

58

"In the Wild" performance

(ODinW13 mean AP (%))

30

35

40

45

LVIS APrare

(%)

100

1k

2k

20k

.1

.2

.3

.4

.5

.6

.7

.9 .8

Self-training only

Fine-tuned on LVIS base

Weight ensemble

OWL L/14 (annotator)

c3

interior

interior

interior

431 bc el

empire 431 bc el

file map athenian

431 bc

file map athenian

431 bc el

file map athenian

map

file bus luxembourg city 3

3

file bus luxembourg city 3

city

3 datei bus luxembourg

city 3

3

3 datei bus luxembourg

bus

dried figs and dates and a glass cup

dried figs and

dried

prunes dried

dried

dried figs and

fresco showing a silver tray containing

prunes dried

fresco showing

dried

monument to josé

canyon for version as of 23

canyon for version as

canyon for version as

39 grand canyon for version as

grand canyon for version as of 23

fichier h with hook

uppercase and lowercase...

version as of

with hook uppercase and lowercase

lowercase for version as of

h with hook uppercase and lowercase

h with hook uppercase and

archivo red script

archivo red script

archivo red script

script palestine flag for version as of

version as of

archivo red script

tram station

tram station

tram station

tram

track system

track system

track system

track system zurich main

station

track system

track system

track system zurich main station

track system

track system

track system

track system

station

file zürich

industriequartier prime...

industrial quarter

file zürich industrial quarter

industrial quarter

and mobimo tower käferberg

industrial quarter

industrial quarter

industrial quarter

industrial quarter

industrial quarter prime and mobimo tower

파일 chikoo sapodilla

november

chikoo

chikoo sapodilla

chikoo

chikoo sapodilla

chikoo

chikoo

chikoo sapodilla

datei datei

impossible map

impossible map

map

september

ramaria

statue mentuhotep aa by

statue mentuhotep

statue mentuhotep

athena promachos

athena

athena promachos

athena

clock

birkenstock

birkenstock

birkenstock

birkenstock

XII

H6

cpl

■

tarp

OWL-ST L/14 self-trained on N-grams, not fine-tuned (Table 1 row 12) lightning rod canteen banner tarp lightning rod pole canteen hose lightning rod water heater tarp pole pole banner canteen banner

apple

peach

pear

date (fruit)

apple

apple

pear

clementine

apple

colander

date (fruit)

cart

pear

pole

apple

colander

cylinder

dropper

apple

pineapple

apple

packet

date (fruit)

apple

water jug

colander

clementine

kiwi fruit

bucket

apple

colander

kiwi fruit

apple

colander

apple

apple

apple

clementine

colander

pole

apple

apple

apple

colander

basket

orange (fruit)

date (fruit)

apple

apple

apple

apple

kiwi fruit

apple

vest

apple

bucket

apple

skullcap

bucket

apple

packet

apple

canteen

basket

hook

basket

bob

apple

thumbtack

bucket

apple

pineapple

elevator car

apple

apple

clementine

apple

cart

apple

canteen

lime

kiwi fruit

pear

apple

turnip

date (fruit)

apple

cart

vending machine

apple

water cooler

apple

apple

apple

packet

colander

apple

banner

orange (fruit)

apple

kiwi fruit

apple

apple

apple

apple

apple

cart

orange (fruit)

clementine

apple

apple

bucket

apple

apple

date (fruit)

banner

date (fruit)

apple

salmon (food)

apple

orange (fruit)

lime

apple

apple

apple

apple

apple

date (fruit)

orange (fruit)

apple

apple

apple

orange (fruit)

cucumber

apple

clementine

thumbtack

basket

apple

thermostat

thumbtack

zucchini

apple

orange (fruit)

apple

tabasco sauce

apple

celery

apple

banana

apple

apple

cart

cart

colander

kiwi fruit

calendar

orange (fruit)

cylinder

apple

basket

orange (fruit)

apple

apple

pineapple

receipt

apple

apple

orange (fruit)

apple

orange (fruit)

apple

lab coat

apple

apple

apple

skullcap

reflector

apple

apple

speaker (stero equipment)

fig (fruit)

turkey (food)

apple

apple

cylinder

plastic bag

date (fruit)

apple

prune

apple

pear

kiwi fruit

dispenser

orange (fruit)

celery

orange (fruit)

cylinder

apple

basket

apple

orange (fruit)

hook

apple

date (fruit)

zucchini

pineapple

pineapple

pineapple

dropper

pineapple

celery

zucchini

orange (fruit)

date (fruit)

cucumber

cucumber

speaker (stero equipment)

grocery bag

cucumber

banana

banana

date (fruit)

cart

thermometer

GALH

tarp canteen

bamboo canteen

basket

pear

banana

apple

basket

date (fruit)

orange (fruit)

apple

speaker (stero equipment)

apple

orange (fruit)

kiwi fruit

basket

bucket

pear

orange (fruit)

basket

orange (fruit)

lightbulb

orange (fruit)

pear

orange (fruit)

apple

potato

apple

banana

apple

signboard

bucket

pear

kiwi fruit

basket

pear

kiwi fruit

basket

kiwi fruit

potato

banana

apple

orange (fruit)

apple

apple

orange (fruit)

apple

kiwi fruit

pear

signboard

pear

kiwi fruit

apple

pear

bucket

banana

kiwi fruit

basket

kiwi fruit

pear

kiwi fruit

apple

apple

kiwi fruit

kiwi fruit

orange (fruit)

kiwi fruit

apple

kiwi fruit

pear

pear

kiwi fruit

orange (fruit)

kiwi fruit

orange (fruit)

apple

orange (fruit)

apple

kiwi fruit

orange (fruit)

pear

kiwi fruit

kiwi fruit

pear

kiwi fruit

orange (fruit)

kiwi fruit

kiwi fruit

pear

orange (fruit)

pineapple

apple

apple

apple

bucket

apple

pear

apple

apple

pear

banana

basket

basket

orange (fruit)

kiwi fruit

kiwi fruit

pear

apple

pear

kiwi fruit

kiwi fruit

pear pear

pear

apple

banana

apple

kiwi fruit

orange (fruit)

kiwi fruit

apple

apple

banana

apple

kiwi fruit

pear

apple

apple

kiwi fruit

pear

pear

kiwi fruit

kiwi fruit

pear

date (fruit)

apple

pear

banana

apple

kiwi fruit

apple

pear

apple

kiwi fruit

apple

apple

apple

apple

apple

pear

apple

kiwi fruit

orange (fruit)

orange (fruit)

bucket

apple

pear

pear

kiwi fruit

kiwi fruit

kiwi fruit

apple

kiwi fruit

orange (fruit)

apple

apple

kiwi fruit

orange (fruit)

apple

kiwi fruit

kiwi fruit

banana

pear

orange (fruit)

banana

apple

pear

apple

apple

apple

plastic bag

kiwi fruit

apple

kiwi fruit

pear

apple

apple

apple

kiwi fruit

orange (fruit)

kiwi fruit

apple

kiwi fruit

bucket

orange (fruit)

pear

kiwi fruit

apple

apple

apple

apple

kiwi fruit

banana

apple

kiwi fruit

apple

basket

apple

orange (fruit)

apple

pear

apple

orange (fruit)

basket

apple

banana

kiwi fruit

apple

apple

pear

apple

basket

kiwi fruit

apple

apple

kiwi fruit

apple

apple

apple

apple

basket

banana

orange (fruit)

apple

apple

apple

apple

apple

orange (fruit)

apple

apple

basket

basket

apple

apple

orange (fruit)

apple

apple

pear

apple

banana

apple

orange (fruit)

thermometer

apple

apple

apple

banana

orange (fruit)

apple

apple

orange (fruit)

orange (fruit)

apple

apple

apple

apple

apple

apple

banana

apple

apple

apple

orange (fruit)

apple

orange (fruit)

banana

apple

apple

apple

shopping cart

orange (fruit)

apple

apple

apple

apple

zucchini

apple

plastic bag

apple

orange (fruit)

apple

apple

apple

apple

orange (fruit)

apple

orange (fruit)

orange (fruit)

kiwi fruit

apple

apple

apple

orange (fruit)

orange (fruit)

apple

apple

zucchini

apple

apple

orange (fruit)

apple

apple

apple

orange (fruit)

apple

apple

orange (fruit)

apple

zucchini

apple

apple

apple

apple

apple

apple

orange (fruit)

apple

apple

apple

zucchini

apple

apple

zucchini

apple

orange (fruit)

pineapple

apple

apple

apple

apple

apple

apple

apple

pineapple

zucchini

apple

pineapple

apple

apple

apple

apple

apple

apple

apple

8

OWL-ST L/14 self-trained on N-grams, not fine-tuned (Table 1 row 12) lightning rod lightning rod lightning rod lightning rod lightning rod lightning rod lightning rod lightning rod lightning rod silo lightning rod lightning rod

lightning rod lightning rod

lightning rod lightning rod

sheep lamb (animal)

black sheep

sheep

fleece

sheep

fleece

sheep

ram (animal)

sheep

black sheep

yoke (animal equipment)

sheep

yoke (animal equipment)

sheep

sheep

yoke (animal equipment)

sheep

sheep

freight car

sheep

yoke (animal equipment)

sheep

sheep

sheep

black sheep

sheep

yoke (animal equipment)

sheep

sheep

sheep

mast

mail slot

sheep

lamb (animal)

yoke (animal equipment)

sheep

sheep

silo

sheep

lamb (animal)

sheep

black sheep

sheep

yoke (animal equipment)

sheep

sheep

black sheep

lightning rod

sheep

sheep

sheep

sheep

black sheep

sheep

sheep

sheep

lamb (animal)

lamb (animal)

mail slot

black sheep

sheep

black sheep

mail slot

sheep

sheep

lamb (animal)

sheep

sheep

lamb (animal)

sheep

sheep

vat

yoke (animal equipment)

sheep

sheep

lamb (animal)

sheep

sheep

lamb (animal)

sheep

sheep

sheep

black sheep

lamb (animal)

sheep

lamb (animal)

lamb (animal)

lamb (animal)

ram (animal)

sheep

sheep

sheep

sheep

mail slot

measuring stick

lamb (animal)

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

black sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

sheep

zucchini

pepper

pepper

hook

folding chair

pepper

pepper

dropper

handle

measuring cup

pumpkin

eggplant

hook

radish

bow (decorative ribbons)

wooden leg

pepper

folding chair

handle

lettuce

bell pepper

clementine

bow (decorative ribbons)

flute glass

cornice

hook

handle

wooden leg

handle

step stool

step stool

chinaware

frying pan

cooking utensil

handle

hook

measuring cup

ironing board

lettuce

cooking utensil

ladle

handle

stool

folding chair

hook

dresser

hinge

pan (metal container)

hairpin

underdrawers

hook

underdrawers

carton

gourd

handle

handle

crossbar

wooden leg

handle

thumbtack

hook

soupspoon

basket

cooking utensil

hook

hook

hook

hook

dixie cup

clementine

hook

carrot

crossbar

underdrawers

hook

cooking utensil

bread

pitcher (vessel for liquid)

wooden leg

cooking utensil

hook

measuring cup

wooden leg

wooden leg

wooden leg

clementine

wooden leg

orange (fruit)

lettuce

underdrawers

eggplant

dixie cup

flower arrangement

folding chair

cooking utensil

cooking utensil

cooking utensil

folding chair

lettuce

cooking utensil

ladle

cooking utensil

cooking utensil

cornice

cooking utensil

cooking utensil

hook

step stool

pan (metal container)

cooking utensil

frying pan

cooking utensil

cooking utensil

frying pan

kettle

frying pan

folding chair

teakettle

cooking utensil

dixie cup

step stool

basket

basket

frying pan

cooking utensil

kitchen table

wooden leg

pan (metal container)

sewing machine

spice rack

cooking utensil

parchment

cayenne (spice)

eggplant

armoire

wooden leg

bread

parchment

lettuce

pan (metal container)

pan (metal container)

wooden leg

bread

pan (metal container)

canteen

gourd

basket

cooking utensil

hook

pumpkin

hook

stool

handle

dresser

hook

gourd

wooden leg

hook

chair

pumpkin

handle

hook

vase

tray

hook

hook

gourd

hook

hook

chair

pot

cooking utensil

cooking utensil

cooking utensil

gourd

hook

frying pan

pan (metal container)

cooking utensil

pottery

stool

stool

cooking utensil

flower arrangement

drawer

cooking utensil

frying pan

frying pan

drawer

cooking utensil

pan (metal container)

frying pan

frying pan

frying pan

frying pan

armoire

eggplant

frying pan

bread

bread

iron (for clothing)

frying pan frying pan

frying pan

pot

table

pot

pan (metal container)

basket

basket

Peer reviews:
comments: SUMMARY:
The paper extends previous OWL-ViT detectors with self-training an 10B Web image-text data. The authors design different self-training strategies, such as token dropping, instance selection and Mosaics augmentation. It is an achievement to beat all previous methods on open-vocabulary benchmarks.

STRENGTHS:
- The authors provide necessary implementation details about scaling up the open-vocabulary detector on massive web data with a self-training method.
- The performance on the open-vocabulary LVIS benchmark is truly impressive.
- The authors promise to open-source their code.

WEAKNESSES:
- GLIP2 performs better than OWL-ST on the ODinW dataset, can the authors explain the possible reasons?
- The authors may consider adding the ablation studies about w or w/o the mosaic augmentation. The additional gain from the mosaic augmentation is not clear.

QUESTIONS:
- What's the meaning of the $AP^{mini}$ metric in Table 1? Does it mean the LVIS test set?

LIMITATIONS:
I like this paper which includes comprehensive experiments and excellent performance, so I think this paper deserves to be accepted. But to me, this paper is not pretty interesting since this paper is more like a technical report with lots of engineering efforts rather than a research paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a self-training recipe for open-vocabulary object detection that leverages weak supervision in the form of image-text pairs from the Web. The authors identify three key ingredients for optimizing the use of weak supervision for detection: choice of label space, filtering of pseudo-annotations, and training efficiency. They propose to use all possible N-grams of the image-associated text as detection prompts for that image and apply only weak confidence filtering to the resulting pseudo-labels. The self-training recipe is applied to the OWL-ViT detection architecture and is called OWL-ST. The authors introduce OWLv2, an optimized architecture with improved training efficiency. The proposed method surpasses prior state-of-the-art methods already at moderate amounts of self-training and achieves further large improvements when scaled to billions of examples. The authors also evaluate their models on a suite of \"in the wild\" datasets and study the trade-off between fine-tuned and open-vocabulary performance.\n\nSTRENGTHS:\nThis paper is well-written, easy to understand, and very rigorous.\nThis paper thoroughly discusses the methods and challenges of extending OWL-VIT to the billion-level image scale.\nThis paper achieves surprising performance.\n\nWEAKNESSES:\nAs the authors mentioned, the biggest issue with this paper is the significant amount of computation required, which makes it difficult for general research institutions to reproduce. Therefore, my question is whether the authors will make the research results available as a foundational model, similar to SAM, for public use. Additionally, can it create a wave of downstream tasks, like SAM did, by providing basic capabilities?\n\nQUESTIONS:\nSince the authors have mentioned SAM, I am curious about how it compares to SAM. Although one is a segmentation model and the other is a detection model, SAM can also be combined with Grounded, such as in https://github.com/IDEA-Research/Grounded-Segment-Anything, to achieve detection. I'm wondering how the authors view the comparison between their model and SAM in terms of certain task capabilities. This is not a necessary question to answer, I just want to have a friendly discussion with the authors.\n\nLIMITATIONS:\nThe main issue is still the enormous amount of computation required. However, if it is open-sourced as a foundational model for public use, it would still make a significant contribution to the community.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper presents OWLv2 model and OWL-ST self-training recipe for open-vocabulary object detection. The detection data is greatly enriched with the aid of self-training. Concretely, the authors use WebLI dataset as the source of weak supervision for self-training. The dataset consists of approximately 10B images and associated alt-text strings (noisy captions). Then OWL-ViT CLIP-L/14 is utilized to annotate all images with pseudo boxes. For self-training at scale, the authors present several techniques including Token dropping, Instance selection and Mosaics to make the training efficient. Experiments on LVIS show the effectiveness of the proposed approach.

STRENGTHS:
1. This paper attemps to scale open-vocabulary object detection with self-training on large-scale weakly labeled dataset. This is a good engineering work built upon the existing detectors and datasets. This work verifies that large model + large data is a good option to enable zero-shot open-vocabulary detection.
2. The experimental results look like promising. Result on LVIS dataset is good.

WEAKNESSES:
1. Authors should carefully polish their paper and fixtypos such as: In L12, L/14->VIT/14; L109, consist->consists.
2. The authors are encouraged to discuss how to handle noise in self-training. What's effect without handling these noise.
3. What's the inference process? The authors are encouraged to describe this in their next version.

QUESTIONS:
See weaknesses box.

LIMITATIONS:
The limitations are adequately discussed in Section 5.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work adopts a self-training strategy to generate web scale object annotations for open-vocabulary object detection. It uses an existing  open-vocabulary object detector (OWL-ViT) to annotate 10B image-text pairs, and uses the annotated dataset for self-training. A various architecture is also proposed to improve the training efficiency. Experimental results show a large performance gain on open-vocabulary detection.\n\nSTRENGTHS:\n1. A self-training strategy to annotate the bounding boxes for the large-scale image-text pairs. This provides an open-vocabulary detection dataset for various location-specific tasks, including detection, segmentation, visual grounding, etc. Despite self-training being widely explored in many previous works (discussed in Sec 2.3), this work provided a detailed ablation in the filtering strategy in Sec. 4.4.\n\n2. The performance is promising in rare classes, indicating the ability of zero-shot transfer with the self-training strategy.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n1. This paper is a coherent work of OWL-ViT. It's not straight forward for me to drop the DETR decoder in OWL, but rather predict a bounding box with each patch token. Such a dense prediction results in a significant number of background proposals (Instance selection in Sec. 3.2), which leads to inefficient detection.\n\n2. While N-grams tend to maintain a higher number of image-text alignments in comparison to rule-based methods (such as highest-scoring or threshold-based approaches), they also tend to generate more redundant alignments, particularly for obvious objects where the confidence scores are higher (e.g., both 'boy', 'a boy', 'a boy in red' might correspond to the same region). Does this bias the model towards learning obvious objects while potentially overlooking the hard examples?\n\nLIMITATIONS:\nThe self-training strategy in the paper requires a large amount of computing resources and web-crawled data, which is not feasible for many colleges and research institutes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}