Open-Vocabulary Object Detection upon Frozen Vision and Language Models
Authors: Weicheng Kuo
Yin Cui
Xiuye Gu
AJ Piergiovanni
Anelia Angelova
Venue: ICLR 2023 poster (2023)
Keywords: open-vocabulary recognition
object detection
vision and language

Abstract: We present F-VLM, a simple open-vocabulary object detection method built uponFrozenVision andLanguageModels.  F-VLM simplifies the current multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining.  Surprisingly, we observe that a frozen VLM: 1) retains the locality-sensitive features necessary for detection, and 2) is a strong region classifier.  We finetune only the detector head and combine the detector and VLM outputs for each region at inference time.  F-VLM shows compelling scaling behavior and achieves +6.5 mask AP improvement over the previous state of theart  on  novel  categories  of  LVIS  open-vocabulary  detection  benchmark. In addition, we demonstrate very competitive results on COCO open-vocabulary detection benchmark and cross-dataset transfer detection, in addition to significant training speed-up and compute savings. Code will be released.


Published as a conference paper at ICLR 2023
F-VLM: OPEN- VOCABULCA OBJECT DETECTION UPON FROZEN VISION AND LANGUAGE MODELS
Weicheng Kuo", Yin Cui!, Xiuye Gul, AJ Piergiovanni', Anelia Angelova" Google Research, Brain Team; 'Google Research, Perception (weicheng, yincui, xiuyegu, ajpiergi, anelia] Bgooglae@goge..coo com
ABSTRACT
We present F-VLM, simple voca-vocyy object detection method built upon Frozen Vision and Language Models. F-VLM simplifies the current multi-stage training pipeline by eliminating the need for knowledge distillation detection- tailored pretraining. Surprisingly, we observe that a frozen VLM: 1) retains the locality-sensitive features necessary for detection, and 2) is strong region clas- sifier. We finetune only the detector head and combine the detector and VLM outputs for each region inference time. F-VLM shows compelling scaling be- havior and achieves +6.5 mask AP improvement over the previous state-of-the-art on LVIS open-vor vocabulary detection benchmark at system level. In addition, we demonstrate very competitive results on COCO open- open-vocabulary detection bench- mark and cross-dataset transfer detection, in addition significant training speed- up and compute savings. The code will be released
INTRODUCTION
Detection is fundamental vision task that aims to localize and recognize objects in an image. However, the data collection process of manually annotating bounding boxes or instance masks is tedious and costly, which limits the modern detection vocabulary size to order of 10a. This orders of magnitude smaller than the vocabulary humans use to describe the visual world. To overcome such limitation, we focus open-vocabulary object detection (Zareian et al., 2021; Gu et al., 2022) to take detection beyond fixed set of vocabulary.
Recently, vision and language models (VLMs) have gained strong open-vocabulary visual recogni- tion capability by learning from Internet-scale image-text pairs (Radford et al., 2021; Jia et., 2021). They are typically applied to zero-shot classification (e.g.. on ImageNet) using frozen weights with- out finetuning, which stands in stark contrast to the existing paradigms of retraining or finetuning when applying VLMs for open- vocabulcyy detection.
Intuitively, in order to align the image content with the text description during training, VLMs may learn locality sensitive and discriminative features that are transferable to object detection. Observations in Figure support our intuition. Surprisingly, features of frozen VI.M contain rich information that are both locality sensitive for describing object shapes (col. 2) and discriminative for region classification (col. 3). This motivates to explore using frozen VLM features for open- vocabulary detection, which entails accurate localization and classification objects in the wild.
We propose F-VLM - simple and scalable open-vocabulary detection approach built upon frozen VLMs. For localization, we simply attach a detector head to predict object regions. For open- vocabulary recognition, we apply the VLM feature pooler (e.g., self-attention layer) on the region features from frozen backbones at test time. We train only the detector head upon a frozen VLM backbone, and combine the detection scores with the corresponding VLM predictions at test time. Our recipe reduces the training complexity of an open-vocabulary detector to below that a stan- dard detector, obviating the need for knowledge distillation, detection-tailored pretraining, or weakly supervised learning. By preserving the knowledge of pretrained VLMs completely, F-VLM main- tains similar philosophy as ViTDet (Li al., 2022c) decouple the detector-specific learning from the more agno-tgi vision knowledge in the backbone.
Project page: http::://hisssooooooccco////////

---

Published as a conference paper at ICLR 2023
Input Image
K-Means Clustering of Frozen Features
GT Regions Classified by Frozen Features
F-VLM (Ours)
We explore the potential of frozen VLM (e.g., CLIP) features for open- vocabulary de- tection. The feature grouping reveals rich semantic and sensiiv information where object boundaries are nicely delineated (col. 2, see Appendix C for more details). The same frozen features can classify groundtruth regions well without finetuning (col. 3). Therefore, we propose to build open-vocabulary detector top of frozen VLM (col. 4) without need for knowledge distillation, detection-tailored pretraining, or weakly supervised learning. F-VLM significantly reduces training complexity and compute requirement, and achieves the state-of-the-arr performance at system level.
Figure 1:
We demonstrate the efficacy of F-VLM on LVIS (Gupta et al., 2019). COCO (Lin al., 2014) and Objects365 (Shao et al., 2019). Here is a summary of our contributions and observations:
We propose F-VIMM simple open-vocabulary detection method upon frozen VLMMs with- out knowledge distillation, detection-tailored pretraining, or weakly supervised learning.
Despite its simplicity, F-VIM achieves strong performance, surpassing the previous state- of-the-art on LVIS open- apen-vocabylary detection benchmark by 6.5 mask AP, at system level and outperforming existing approaches in cross-dataset transfer (COCO, Ohjects365).
F-VLM shows compelling scaling behavior with consistent performance improvements by increasing the backbone capacity (e.g., +14.2 LVIS mask AP, with our largest backbone).
F-VLM has much fewer trainable parameters, allowing it to train significantly faster. Com- pared with a strong open- vocabblaly detection method ViLD (Gu et al., 2022), F-VLM not only achieves better performance, but also provides up to 200 training compute savings.
We hope these findings will facilitate the research community to further explore frozen VLMs for broader range computer vision tasks.
RELATED WORK
visual recognition and representation learning. Zero-shot and open-vocabulary recognition has been a long-standing problem in computer vision. Earlier works use the visual attributes to represent categories binary codebooks and learn predict the attributes for novel categories (Jayaraman & Grauman, 2014; Rohrbach et al., 2011). DeViSE (Frome et al., 2013) and ConSE (Norouzi et al., 2014) pioneer to learn a joint image-text embedding space using deep learning. Many works have shown the promise of representation learning from natural language associated with images, such as image tags (Chen Gupta, 2015; Divvala et al., 201 4; Joulin al., 2016) or text descriptions (Desai & Johnson, 2021; He & Peng, 2017; Sariyildiz et al., 2020; Wang et al., 2009; Zhong et al., 2021). Recently, popular large VLMs scale up by training on billions of image-text pairs and acquire strong image-text representation by contrastive learning (Radford et al., 2021; Jia et al., 2021; Pham et al., 2021; ;; Zhai et al., 2022). These models achieve strong zero-shot performance many classification benchmarks and show clear benefits in scaling model capacity.
Zero-shot/Oppen--ccablll
While all the above works study image-level recognition, the focus of this paper is on the object- level understanding. Recently, Vasconcelos al. (2022) has shown frozen classification models
2

---

Published as a conference paper at ICLR 2023
beneficial for closed-occablarry detection with adequate detector head capacity. In addition, frozen VLM can serve as a teacher model and combine with self-training for zero-shot semantic segmentation (Zhou et al., 2022a). In contrast, we study how to use frozen VLM directly part of an open-vocabulary object detector.
object detection. It is costly and labor-intensive to scale up data collection and annotation for large vocabulary detection. Zero-shot detection aims to alleviate the challenge by learning to detect novel categories not present in the training data. Many methods address this by aligning the image region features to category word embeddings (Bansal et al., 2018; Rahman et al., 2020; Demirel et al., 2018; Zheng al., 2020), or synthesizing visual features with a generative model (Hayat et al., 2020; Zhu et al., 2020). Recently, Zareian et al. proposes the open-vocabulary detection (OVD) benchmark with view bridge the performance gap between ZSD and supervised learning (Zareian et al., 2021). The model was first pretrained on image-caption data to recognize novel objects, and then finetuned for zero-shot detection (Zareian et al., 2021).
Zerr-Shoo/Openn-vcaallla
Following the OVD benchmark, ViLD (Gu et al., 2022) proposes to distill the rich representation of pretrained VLM into the detector, and DetPro (Du et al., 2022) improves upon ViLD by ap- plying the idea of prompt optimization. RegionCLIP (Zhong et al., 2022) develops region-text pretraining strategy that leverages pretrained VLMs and amage-caption data, while Detic (Zhou et al., 2022c) jointly trains a detector with weak supervision. VL-PPLM (Zhao et al., 2022) explores labulo-la on unlabeled data with object proposals and VLMMs for OVD. GL.PP (Li et al., 2022b) formulates object detection as phrase grounding task and pretrains on a wide variety of detection, grounding, and caption datasets for zero/few-shot object detection. Similarly, OWL-ViT (Minderer et al., 2022) proposes to finetune pretrained vision transformers datasets. All mentioned methods require training the entire detector from scratch, finetuming after detection-tailored pretraining, or training on suite of F-VLM trains only the standard detector head upon a frozen VLM without using any of the above.
a suite of detection/grounddnng
datasets. In contrast,
3 METHOD
3.1
OVERVIEW
In this paper we address the problem of open-vocabulary object detection. At training time, the model has access to the detection labels of CB base categories, but needs to detect objects from a set of ON novel categories at test time. To make the settings more practical (Zareian al., 2021), we follow previous works and assume the availability of a pretrained vision and language model (VLM) which has learned from plenty of image-text pairs on the internet (Gu et al., 2022).
Figure 2 shows the overall F-VLM architecture, We propose to build the open-vocabulay object de- tector upon frozen VLMS by training only the detector head upon frozen features, which guarantees to completely preserve the vocabulcay classification ability of pretrained VLMs.. At test time, we combine the detector scores with the VLM scores to obtain open-vocabulary object detection scores. By directly using frozen pretrained models, our approach is simple and easily scalable.
3.2 PRETRAINING FROM VISION AND LANGUAGE MODELS
Recently, Vision and Language Models (VLM) are popular because of their rich knowledge and strong representation for both visual and linguistic domains. We desire retain their knowledge as much as possible, in order to minimize the effort/cost to adapt the VLMs for open-vocabulary detection. Following existing works (Du et al., 2022: Gu et al., 2022; Zhong et al., 2022), we focus on contrastively pretrained VLMs in this paper egg. (Jia et al., 2021; Radford et al., 2021). Contrastive VLMs typically have the image and text encoders trained jointly with contrastive objective. We use the frozen image encoder as the detector backbone, and the text encoder for caching the text embeddings of detection dataset vocabulary offline (see Sec. 3.3).
We consider the VLM image encoder in two parts: 1) the feature extractor F(.), e.g. ResNet- 50 (Radford et al., 2021), and 2) the last feature pooling layer P(.), e.g. an attention pooling layer (Radford et al., 2021). We adopt the same backbone architecture as the image feature extractor F(-). which allows us to directly use the frozen weights and inherit the rich semantic knowledge (see Fig. 2a). Along with the backbone initialization, we also adopt the same image pre-processing scheme as the VLM pretraining to maintain its open- vocabulayy recognition ability. We use the last VLM pooling layer P(.) for open-vocabulary recognition test time only (see Sec. 3.4). Build-
3

---

Published as a conference paper at ICLR 2023
Input Image
Detection Losses
Legend
Loss
Detection Boxes & Masks
Loss
Outputs
Protrained Image Encoder
Detector
Trainable
Detection Scores fr
Frazen
Fgy fg
Base Categories
Pretrained Text Encoder
Detection Region Embed.
Person
Clexs Teut Embedd.
(a) F-VLM training architecture. At training time, F-VLM simply detector with the last classification layer replaced by base-category text embeddings. The detector head is the only trainable part of the system, which includes RPN (Ren et al., 2015), FPN (Lin et al., 2017), and Mask R-CNN heads (He et al., 2017).
Top-lovel
Map
Sy Seel Spen.
Input
Detection
Region Proposals b
ROY ALIGN
VLM Pooling Layer
VLM Scores M
Boses
Protrained Image
Detector Head
Detection Scores
Open Detaction Scores
Mean [see
Base Categories
Ereu v..
Pretrained Text Encoder
Cat
Legand
VLM
Oetection
Class finbed.
Noxel Closs Embed.
Novel
Embrdd
Region Embbed
Roat
Outputs
Frozen
Ug My
Fl
by Fn
Text Frare
(b) F-VLM inference architecture. At test time, F-VLM uses the region proposals to crop out the top-level features of VL.M backbone and compute the VI.M score per region. The trained detector head provides the detection boxes and masks, while the classification scores combination of detection and VLM scores.
Figure 2: F-VLM architecture. We present both training and inference time architectures of F- VLM, where the VLM pooling layer and detection score combination are the differences.
ing upon the frozen backbone features, we adopt Mask R-CNN (He et al., 2017) head and feature pyramid network (Lin et al., 2017) as the detector head following previous works (Du et al., 2022; Gu et al., 2022; Zhong et al., 2022). The detector head is randomly initialized and is the only trainable component of F-VLM. Despite the image-level pretraining, we found empirically that the frozen VLM backbone contains adequate locality sensitive features to enable accurate downstream detection (see Appendix C).
3.3 TEXT-EMBEDDING REGION CLASSIFIER
Notations: Let's define I as the input image, F(I) the backbone features from the image encoder. Let @(-) be the function that yields region embedding rs from F(I) and a given box region proposal b, which involves FPN (Lin et al., 2017), ROI-Align (He et al., 2017), and Faster R-CNN head (Ren et al., 2015). We have:
r5 = Q(F(1))))
(I)
Standard detectors use K-way classifier because the training and test time categories are the same. This design does not support the open-vocabulary settings which require new categories to be added at test time. To accommodate this, we replace the last fully connected layer with the text embeddings of base categories (see Fig. 2a). At inference time, we can simply expand the text embeddings to include novel categories for open- open-vocabulary detection (see Fig. 2b). An advantage of such design that the system can generalize to the novel categories near CB in the embedding space.
To generate the text embeddings, it is critical to use the matching text encoder of the image encoder because they were pre- pre-trained together. Apart from CB. the background category is represented by a generic phrase "background" for compatibility with other categories. At training time, the proposals not matched to any groundtruth boxes in Cn are treated as background. For each region, we compute the cosine similarity of rs, with the text embeddings of Cn and "background"; and apply learnable
4

---

Published as a conference paper at ICLR 2023
temperature on the logits. The detection scores z(rb) are given by:
z(r,) = Softmar [oos(r,, tbg), cos(rb; t1), ... cos(rs, t(c=))))
(2)
where cas b) a aT// b/ /|||a|]||bb]) and t denotes the text embeddings of class i. We apply the standard softmax cross entropy loss on the logits (see Fig. 2a). At test time, we keep the "back- ground" category and expand the text embeddings from CB to Cg U Cn for open-vocabulary de- tection. Similar designs have been used by previous works (Zareian et al., 2021; Gu et al., 2022).
3.4 OPEN-VOCABULAY RECOGNITION
The ability to perform open-vocabulary recognition at region level integral F-VLM. Since the backbone features are frozen, they do not overfit to the base categories and can be directly cropped for region-level classification. F-VLM performs this open-vocabulary classification only at test time.
To obtain the features for a region b, we apply the VLM pooling layer P(-) on the cropped backbone output features F(T) (see Sec. 3.2 for notations). Because the pooling layer requires fixed-size inputs, e.g. 7x7 for R50 (Radford ct al., 2021), crop and resize the region features with ROI-Align R()) (He et al., 2017) (see Fig. 2b). Unlike cxisting works (Gu et al., 2022; Du et al., 2022), we do not crop and resize the RGB image regions and cache their embeddings but train the detector head in one stage. This is simpler and more space-efficient. In addition, we do crop VLM region features with R(-) during training because the backbone features are frozen. Using the notations from equation 1, we obtain the VLM region embedding vo by:
separate offine process,
V, = P(((F(()))
where b denotes the box region and va corresponds U1...., Uk in Fig. 2b. Note R(-) is used at test time only. Similar to equation 2, we compute the VLM scores by cosine similarity as follows:
w(v)) Softmaz( _
[cos(Vb, Vv, tbg), cos( (vs, t1), "., cos(vs, CBuuN)]
(4)
T
where 7' is fixed temperature and the text embeddings include both the Cß and Cn at inference time (see Fig. 2b). We use a fixed temperature to adjust the scale of VLM scores relative to the detection scores in equation 2. In the special case when the region b is equal to the whole image, the VLM scores w (vb) becomes equivalent to the zero-shot image classification scores.
Despite never being trained regions, the cropped region features of F(-) maintain good open- vocabulary recognition ability. However, we observe the cropped region features are not sensitive enough to the localization quality of the regions, i.e. loosely vs tightly localized box both have similar features. This may be good for classification, but is problematic for detection because we need the detection scores reflect localization quality as well. To remedy this, we apply the ge- ometric mean to combine the VLM scores w (v(v);; in equation 4 with the detection scores a(r)); in equation for each region b and category i. The final detection scores (Tr)); are given by:
(Tr); (2(rr)( w(v)); if Cn =((6)) (1 (1-8) w(v))? ifie
where a, 3 [0, control the VLM score weights for base/novel categories, and the background score comes directly from the detector iee., s(rr)o = 2(r))o- Compared to the ensemble system in (Gu al., 2022), our design is simpler without need for knowledge distillation or double Faster R-CNN heads. We show ablations of different score fusion designs in Appendix A.2.
3.5 OPEN- VOCABUOCYA LOCALIZATION
How to localize and separate the novel objects from the background is an important problem in open- vocabulary detection. Standard detectors not designed for localizing novel objects because most of them apply class-specific localization, including the box regression and mask prediction heads, e.g., Mask R-CNN (He et al., 2017). Inspired by the learned objectness (Kim et al., 2022; Kuo al., 2015: Wang et al., 2020), we use class class-agnostic box regression and mask prediction heads instead. That is, for each region proposal, we predict one box and one mask for all categories, rather than one
5

---

Published as a conference paper at ICLR 2023
per category. This simple change allows us to localize novel objects in the open-vocabulary settings. We note that F-VLM framework is not specific to the choice of Mask R-CNN detector head and other models can potentially be applied as well e.g. (Carion et al., 2020; Redmon et al., 2016). We choose Mask R-CNN per existing works (Gu et al., 2022; Zareian et al., 2021; Zhong et al., 2022).
EXPERIMENTS
Implementation Details. We choose Mask R-CNN (He et al., 2017) with feature pyramid net- work (Lin al., 2017) as our detector head throughout the paper. The head design follows (Ghiasi et al., 2021; Gu et al., 2022). We train the model for 46.1k iterations with 1024x1024 image size, large scale jittering (Ghiasi al., 2021), batch size 256, weight decay le-4, momentum 0.9, and an initial learning rate 0.36. For the score combination, we use a = 0.35 and f = 0.65 in equation 5. = 0.01 in equation We
We use maximum of 300 detections per image, and set temperature use CLIP (Radford et al., 2021) prompt templates and take the average text embeddings of each category. Please refer to Appendix G for a full list of hyper- parer-pr configurations.
4.1 OPEN- DETECTION BENCHMARK
LVIS Benchmark. We evaluate our approach on the LVIS dataset (Gupta et al., 2019) which con- tains a large and diverse set of 1203 object categories suitable for open- vocabulary detection. Fol- lowing the existing works (Gu et al., 2022; Zhong et al., 2022), treat the frequent and common categories as the base categories C3 for training. and hold out the rare categories as novel categories CN for testing. Mask AP, is the main metric we benchmark on. To ensure reprodueibility. we report the mean of independent runs following the protocol of (Gu et al., 2022) and the best practice of LVIS challenge (Gupta et al., 2019). For fair comparison, we adopt the same Mask R-CNN head architecture as (Gu et al., 2022) and use the same large scale jittering recipe (Ghiasi et al., 2021).
Table 1 presents our results on LVIS. In the R50 comparison, F-VLM ranks second among the other alternatives based on knowledge distillation, pretraining, or joint training with weak supervision. The leading DetPro (Du et al., 2022) shows the effectiveness of prompt optimization (Zhou al., 2022b) which can potentially benefit F-VLM too. In the system-level comparison, we observe the performance of F-VLM scales up nicely with frozen model capacity, even though the amount of trainable parameters is fixed. Our best model achieves 32.8 AP,, which is +14.2 AP, from the R50 baseline and the best published results on this benchmark to our knowledge. Compared to the best existing approach (ViLD-EN-B7), we outperform by 6.5 mask AP, the novel categories (and +5.6 overall mask AP). We provide additional results using standard Ix/3x training recipes (Wu et al., 2019) in Appendix B, where F-VLM shows similarly strong performance on shorter recipes and smaller batch size by using frozen backbones.
COCO Benchmark. Many existing works zero-shot detection (Bansal et al., 2018) and open- vocabulary detection (Zareian et al., 2021; Gu et al., 2022; Zhong et al., 2022) benchmark on COCO. This setup divides COCO vocabulary into 48 base categories for training and 17 novel categories for testing. We follow the standard practice and report results in the generalized detection settings without instance segmentation. The main metric is AP50 of novel categories. Similar to LVIS, we report the mean of 5 independent runs to ensure reproducibility.
Due to the smaller number of training categories, we observe tendency to overfit when we re-use the same LVIS training recipe. F-VLM does not rely on additional objectives e.g. knowedlge distil- lation or weak supervision to counter-balance overfitting. We therefore reduce the training epoch, background weight, and increase the weight decay to mitigate this. Please refer to Appendix G for a full list of hyper-parameters
Table 2 shows that F-VLM is very competitive among the published results. Compared to the leading RegionCI.IP (Zhong et al., 2022) which uses additional caption pretraining, F-VLM directly uses a frozen CLIP backbone. In fact, F-VLM significantly surpasses the CLIP-R50 pretrained version of RegionCLIP, which does not leverage pretraining on caption data. Compared to other approaches, F- VLM offers better performance without the use of secection-tailoreed pretraining, weakly supervised learning, or knowledge distillation.
Training Resource Benchmark. We explore the benefits of frozen VLMs in terms of training resource savings. We benchmark with ViLD (Gu et al., 2022) as it is most comparable to F-VLM. Both adopt the same Mask R-CNN head configuration and training recipe (Ghiasi al., 2021),

---

Published as a conference paper at ICLR 2023
Table I: LVIS Open-Vocabulary Object Detection Benchmark. F-VLM outperforms the best existing approach by 6.5 mask AP on novel categories. All methods use the same instance-level supervision from LVIS (Gupta al., 2019) base categories, CLIP (Radford al., 2021) pretraining, and fixed prompt templates unless noted otherwise. t; Pretraining with CC-3M (Sharma et al., 2018). 4: Prompt optimization (Zhou et al., 2022b) and SoCo pretraining (Wei et al., 2021). *: Joint training with IN-21k (Deng et al., 2009). *: ALIGN model (Jia et al., 2021).
Pretrained CLIP
Trainable Backhone
Backbone (# Params)
Method
Distill
AP.
AP
R50 Comparison:
R50
ViT-B/32
ViLD (Gu et al., 2022)
16.1
22.5
R50
VIT-B/32
ViLD-Ens, (Gu et al., 2022)
16.6
25.5
R50
VIT-B/32
DetPro (Du et al., 2022)
/
19.8
R50
VIT-B/32
Detic-ViLDD (Zhou et al., 2022c)*
X
17.8
26.8
R50
R50
RegionCIIIP (Zhong al., 2022)
17.1
28.2
R50
R50
F-VLM (Ours)
18.6
System-level Comparison:
R152 (60M)
ViT-B/32
ViLD (Gu et al., 2022)
18.7
23.6
R152 (60M)
ViT-B/32
ViLD-Ens, (Gu al., 2022)
18.7
26.0
EN-B7 (67M)
ViT-L/14
ViLD- Ens. (Gu al., 2022)
21.7
29.6
EN-B7 (67M)
EN-B7"
ViLD-Ens. (Gu et al., 2022)
26.3
29.3
R50 (26M)
ViT-B/32
DetPro-Cascade (Du al., 2022)²
20.0
27.0
R50 (26M)
VIT-B/32
Detic-CN2 (Zhou et al., 2022c)'
24.6
32.4
R50x4 (87M)
R50x4
RegionCIIIP (Zhong al., 2022)
22.0
32.3
ViT-L/14 (303M)
ViT-L/14
OWL-ViT (Minderer et al., 2022)
25.6
34.7
R50x4 (87M)
R50x4
F-VLM (Ours)
R50x16 (167M)
R50x16
F-VLM (Ours)
X
30.4
32.1
R50x64 (420M)
R50x64
F-VLM (Ours)
X X
32.8
34.9
Table 2: COCO Open- Vocabulayy Object Detection Benchmark. F-VLM is very competitive with the other methods trained with various sources, All methods use the ResNet50 backbone (He et al., 2016; Radford et al., 2021). RegionCLIP additionally use COCO Captions! (Lin et al., 2014) or CC3M (Sharma al., 2018) for pretraining. * CLIP initialization without region-level pretrain-
ing. * Joint training with COCO captions.
Method
Training source
Novel AP
AP
WSDDN (Bilen & Vedaldi, 2016)
19.7
19.6
image-level labels in Cp UC
Cap2Det (Ye et al., 2019)
20.3
20.1
ZSD (Bansal et al., 2018)
0.31
24.9
DELO (Zhu et al., 2020) PL (Rahman et al., 2020)
instance-level labels in C'g
3,41
13.0
4.12
27.9
image captions in Cn U CN
OVR-CNN (Zareian et al., 2021)
22.8
39.9
instance-level labels in C'
CLIP-RPN (Gu al., 2022)
26.3
27.8
ViLD (Gu et al., 2022)
27.6
51.3
Detic* (Zhou al., 2022c)
27.8
45.0
CLIP image-text pairs
RegionCLIPI (Zhong et al., 2022)
instance-level labels in Cp
31.4
50.4
RegionCI.IP (Zhong et al., 2022)
26.8
47.5
RegionCLIP* (Zhong et al., 2022)
14.2
42.7
F-VLM (Ours)
28.0
39.6
and neither require detection-tailored pretraining. We follow ViLD and compare the training cost on TPUv3 cores on the same batch size. The data about ViLD training time and resource use is obtained directly from the authors (Gu et al., 2022). To keep the benchmark simple, assume the pretrained VLMs are given and exclude their training costs from the comparison. For F-VLM, we the R50x64 backbone and report the average 5 independent runs.

---

Published as a conference paper at ICLR 2023
Table 3: Training Resource Benchmark. We report LVIS mask AP, show the performance vs
training cost trade-off. F-VLM can outperform ViLD (Gu et al., 2022) with 226x less compute.
Training Cost (Per-Core-Hour)
Training Cost Savings
Method
Mask AP,
#lters
Epochs
ViLD-EN-B7 (Gu et al., 2022)
26.3
180k
460
8000
1x
F-VLM (Ours)
32.8
46.
118
565
14x
F-VLM (Ours)
31.0
5.76k
14.7
71
113x
F-VLM (Ours)
27.7
2.88k
7.4
226x
Table shows that F-VLM can achieve top performance with much less compute. Compared to the state-of-the-art (ViLD-EN-B7)
system level, F-VLM can achieve better performance with only 7.4 epochs of training, which is 226x more compute-efficient and x faster in wall clock time. We believe the efficiency gain arises from the frozen backbone, which substantially simplifies the learning process. This is orthogonal to the detection-tailore pretraining used by existing works to speed up training (Zareian et al., 2021; Zhong et al., 2022). Apart from resource savings, F-VLM has potential for substantial memory savings at training time by running the backbone in inference mode (see Appendix E for more details). F-VLM system runs almost as fast standard detector (He et al., 2017) at inference time, because the only addition is a single attention pooling layer (Radford al., 2021) on the detected region features (see Fig. 2b).
Transfer Detection Benchmark. We explore the potential of F-VLM as a general-purpose detector for different data sources with view to move towards trained on one dataset can be directly applied to another by swapping out the vocabulary without any finetuning, e.g., replacing the 1203 LVIS categories with COCO 80 categories. The models we use are trained on LVIS base categories and tested on COCO and Objects365-v1 validation splits following the transfer setup of ViLD (Gu et al., 2022). Since COCO and Objects365 have smaller vocabularies than LVIS, category and image overlaps are hard to avoid. We calculate the vocabulary overlap between COCO/Ohjects365 and LVISS base categories to be 91% and 63% respectively. Please refer to Appendix D for more discussion about this benchmark.
dataset-specific detection. F-VLM
Table 4 presents the results in comparison with prior works and supervised baselines. We observe the performance of F-VLM improves steadily as we scale up the frozen model capacity. On Ob- jects365/COCO, the best F-VLM outperforms existing works ViLD by +3.2/45.9 and DetPro by +4.9/45.6, closing the gap with supervised model on COCO (-33%) and Objects365 (-40%). The results are reported in Box AP averaged over runs. There is no longer distinction between base and novel categories in the transfer setting, we assume all categories
novel and use 3 alone to combine detection and VLM scores in equation 5 (see Sec. 3.4). We observe that only the detection scores are needed (3 0) for COCO, while the optimal 3 0.3 0.4 on Objects365.
Table 4: Transfer detection of F-VLM. We evaluate LVIS-trained F-VLM COCO and OB- jects365 without finetuning. F-VLM demonstrates strong scaling property with a gain of +7.3/+5.8 AP on COCO/Objects365 by increasing backbone capacity.
COCO
Objects365 APso
Method
AP
APs0
AP75
AP
AP75
Supervised (Gu et al., 2022)
46.5
67.6
50.9
25.6
38.6
28.0
ViLD-R50 (Gu et al., 2022)
36.6
55.6
39.8
11.8
18.2
12.6
DetPro-RS5 (Du et al., 2022)
34.9
53.8
37.4
12.1
18.8
12.9
F-VLM-R50 (Ours)
32.5
53.1
34.6
11.9
19.2
12.6
F-VLM- -5004 (Ours)
36.0
57.5
38.7
14.2
22.6
15.2
F-VI.M-550x 16 (Ours)
37.9
59.6
41.2
16.2
25.3
17.5
F-VI.MRR50x64 (Ours)
39.8
61.6
43.8
17.7
27.4
19.1
4.2 ANALYSES AND VISUALLIZATION
Ablations. We present ablation studies on backbone finetuning, score fusion design/parameters, feature pyramid capacity, and background weight in Appendix A. Here summarize findings.
In the exploration of finetuning vs frozen backbone (see Table 5), we discover that finetuning im- proves the standard detection (base categories) but slightly hurts the open-vocabulary detection
8

---

Published as a conference paper at ICLR 2023
(novel categories). It remains open question whether more sophisticated finetuning strategies can benefit open- vocabulayy detection.
In the score fusion studies (see Table 6), observe that geometric mean is significantly better than the arithmetic mean (+8 AP,). It is likely because the geometric mean requires a high-scoring region to have good detection and VLM scores simultaneously, whereas the arithmetic mean may favor regions with high detection or VLM scores. In Table 7a and Table 7b, we study the score fusion weights and observe that detection nor VLM scores alone are sufficient as 3 = 0, both yield sub-optimal performances. In Table 7c, we study the temperature equation and find the optimal T 10-2 is much smaller than the value of learnable x 1.0 at the end of training (see equation 2). This highlights the need to use separate
0.65 and
0.35 are most beneficial (see equation 5). Neither
VI.M scores instead of using T for both detection and VI.M scores.
In Table 8, we explore the effects of increasing feature pyramid capacity to enhance the represen- tation learned upon the frozen backbone features. Our results show that larger pyramid benefits standard detection (base categories) without compromising the open-vocabulary detection (novel categories). In Table 9, we study the influence of background weights by Zareian et al. (2021); Zhong et al. (2022) on open-vocabulary detection and found it to help slightly (0.1 to 0.5 AP,). Please refer to Appendix A for the full experimental results.
Detection Visualization. Figure visualizes F- F-VLM open-vocabulary detection and transfer detection tasks. The transfer detection is done by replacing the dataset vocabulary without finetun- ing. On LVIS and Objects365 (Shao et al., 2019), F-VLM correctly detects both novel and common objects. Please see Appendix H for more details and visualization.
A key benefit of open- vocabubuyy detection is To test on distriitittiittoonnn data with categories given by users on the fly. Thus, we apply F-VLM to Ego4D (Grauman et al., 2022), a real-world ego-centric application. Despite the large domain shift, F-VLM is able detect many novel and common objects. Please see Appendix for more details and visualization.
LVIS Novel Categories
Objects365 Transfer
Ego4D Transfer
BAR
Oresding
50:54
Seiyshi lra
Twcbel
5op
Mancke
Figure 3: F-VLM open-vocabulary and transfer detections. 1-2nd col.: Open- vocabulary de- tection on LVIS. We only show the novel categories for clarity. 2-4th col.: Transfer detection on Objects365. 4-6th col.: Transfer detection on Ego4D. Novel categories detected: fedora, martini, pennant, football helmer (LVIS); camel, slide, goldfish (Objects365), exit sign, recycle bin, window, soy sauce, wooden basket, cereal, bag of cookies, instant noodle, salad dressing. ketchup (Ego4D).
CONCLUSION
We present F-VLM simple open-vocabulary detection method built upon frozen VLMs without a need for knowledge distillation, iiteeciionttallrre pretraining, or weakly supervised learning. F. VLM offers significant training speedup and compute savings, achieves the new state-of-the- LVIS benchmark at system level, and shows very competitive transfer detection. We hope this study help the community explore frozen VLMs for wider range of vision tasks.

---

Published as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We plan to open source the code for reproducibility. We have provided the model, experimental and implementation details in the paper or the supplemental materials (Section 4 and Section G). The CLIP model (Radford et al., 2021), Mask R-CNN model (He et al., 2017), and all datasets used (Lin al., 2014; Gupta et al., 2019; Shao et al., 2019; Grauman al., 2022) this work are publicly available.
ETHICS STATEMENT
We demonstrate capabilities in detecting previously unseen categories of objects, and partic- ularly on challenging benchmarks and transfer settings. Our models utilize the rich information embedded in Vision-Language Models, which may reinforce deficiencies and biases the internet data and propagate potentially harmful biases or stereotypes. The models we trained are used here for evaluation/benchmark purposes and need more rigorous probing for bias, fairness, etc., before using them for any other purpose.
ACKNOWLEDGMENTS
We thank colleagues at Google Research for their advice and helpful discussion.
REFERENCES
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV, 2018.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. CVPR, 2016.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In ICCV, 2015.
Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinbis. Zero-shot object detection by hybrid region embedding. BMVC, 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly- supervised visual concept learning. In CVPR, 2014.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, 2022.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NeurlPS, 2013.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In CVPR, 2021.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray,
10

---

Published as a conference paper at ICLR 2023
Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jáchym Kolář, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Rujjic Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawa- har, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torre- sani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, June 2022.
Xiuye Gu, Tsung-Y. Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2022.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance seg- mentation. In CVPR, 2019.
Nasir Hayat, Munawar Hayat, Shafin Rahman, Salman Khan, Syed Waqas Zamir, and Fahad Shah- baz Khan. Synthesizing the unseen for zero-shot object detection. In ACCV, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Xiangteng He and Yuxin Peng. Fine-grained image classification via combining vision and lan- guage. In CVPR, 2017.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR, pp. 17980-17989, June 2022.
Dinesh Jayaraman and Kristen Grauman. Zero shot recognition with unreliable attributes. In NeurIPS, 2014.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le. Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021.
Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual fea- tures from large weakly supervised data. In ECCV, 2016.
Dahun Kim, Tsung-Y. Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open- world object proposals without learning to classify. IEEE Robotics and Automation Letters, 2022.
Weicheng Kuo, Bharath Hariharan, and Jitendra Malik. Deepbox: Learning objectness with convo- lutional networks. In ICCV, 2015.
Junnan Li. Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: image pre-training for unified lingunelanguage understanding and generation. arXiv preprint arXiv:2201. /2086, 2022a.
Bootstrapping language-
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022b.

---

Published as a conference paper at ICLR 2023
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In ECCV, 2022c.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detec- tion with vision transformers. In ECCV, 2022.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. 2014.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer learning. CoRR, abs/2111 01//211..1000 100000 2021. URL https://arxiv.. wwwaaaiiio.rr...eaeeeooeee www.oorgg//b/aaeoooeeeo.os.eee/e 2atss./11...1.. 10050.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ICMI., 2021.
Shafin Rahman, Salman Khan, and Nick Barnes. Improved visual-semantic alignment for zero-shot object detection. In AAAI, 2020.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Shaoqing Ren, Kaiming He, Ross B Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Mareus Rohrbach, Michael Stark, and Bernt Schiele. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In CVPR, 2011.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, 2020.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, pp. I 11838-15650, 2022.
Cristina Vasconcelos, Vighnesh Birodkar, and Vincent Dumoulin. Proper reuse of image classifica- tion features improves object detection. In CVPR, 2022.
Josiah Wang, Katja Markert, Mark Everingham, et al. Learning models for object recognition from natural language descriptions. In BMVC, 2009.
Rui Wang, Dhruv Mahajan, and Vignesh Ramanathan. What leads to generalization of object pro- posals? In ECCV, 2020.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.1094 2021.
12

---

Published as a conference paper at ICLR 2023
Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. In NeurIPS, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https:
Keren Ye, Mingda Zhang, Adriana Kovashka, Wei Li, Danfeng Qin, and Jesse Berent. Cap2det: Learning to amplify weak caption supervision for object detection. In ICCV, 2019.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, Anastasis Stathopoulos, Manmohan Chandraker, Dimitris Metaxas, et al. Exploiting unlabeled data with vision and language models for object detection. In ECCV, 2022.
Ye Zheng, Ruoran Huang, Chuanqi Han, Xi Huang, and Li Cui. Background learnable cascade for zero-shot object detection. In ACCV, 2020.
Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li. Learning to generate graph from natural language supervision. In ICCV, 2021.
Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Lu- owei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region- basodd language- image pretraining. In CVPR, 2022.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022b.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. ECCV, 2022c.
Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Don't even look once: Synthesizing fea- tures for zeoo detection. In CVPR, 2020.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A ABLATION
A.1 FINETUNING VERSUS FROZEN BACKBONE
We explore the pros and cons of backbone finetuning compared to the frozen backbone. We observe that finetuning the backbone with the same training recipe diverges, so we apply gradient clipping (max norm 1.0) and reduce the backbone learning rate significantly. Table 5 shows that although finetuning can benefit the base categories, slightly compromises the novel category with higher memory/compute footprint.
Table 5: Finetuning frozen backhone. Finetuning does not benefit the novel categories (AP,) but improves the base categories (APc, AP,).
Backbone LR
AP,
AP.
AP,
AP
lc-3
18.1
25.7
30.2
26.2
le-4
18.1
24.9
28.8
25.3
0.0
18.6 (+0.5)
24.0
26.9
24.2
A.2 SCORE FUSION
We explore the use of arithmetic vs geometric means to fuse the VLM and detection scores in equa- tion 5. Table shows that using geometric mean is significantly better by more than points. In Figure 4, we perform dense grid sweep over a, B and confirm the 8 8-point gap between geometric and arithmetic means still holds.
Arithmetic (T=0.02)
Geometric Mean (T=0.01)
2-0.5
#-0.65
p=:08
(H0.45
140.55
1+0.85
j-1
071.16
070.25
4.7033
0-0006
6-0.75
0-00 15
m=0.225
@+0.35
e-0 05.55
0=0.75
Figure 4: Hyper-parameter sweep score fusion parameters. We observe that geometric means (right) are significantly better than arithmetic means (left). All results are based on a trained F-VLM R50 model.
We perform a more in-depth study of score fusion parameters in Table From the table, we see that B is the main tunable parameter of model, and the performance is relatively robust to For most practical use cases, we recommend setting = 0.01. The temperature 7 in Equation is learned automatically and needs tuning.
A.3 FEATURE PYRAMID CAPACITY
We explore the effects of increasing the feature pyramid (Lin et al., 2017) capacity to enhance the representation learned upon the frozen backbone features. To increase the FPN capacity, we simply repeat the lateral and top-down connections of FPN N times before applying the post-hoc convolution. We insert a ReLU and BatchNorm layer in each lateral connection, and add a skip connection from the backbone feature maps to every level (i.e. 2, 2...... N). The post-hoc convolution and BatchNorn layers are kept the same. All runs are repeated 5 times and trained for 46. steps following the same protocol the LVIS benchmark of the manuscript.
14

---

Published as a conference paper at ICLR 2023
Table Score Fusion. We study different score fusion mechanisms of F-VLM. We report AP, and AP on LVIS. Geometric mean is significantly more effective than arithmetic mean. All results are average over independent runs using R50 backbone.
Fusion
a
T
AP,
AP
Arithmetic
0.65
0.35
0.01
9.1
16.4
Arithmetic
0.65
0.35
0.02
9.3
19.8
Arithmetic
0.5
0.75
0.02
10.3
15.9
Geometric
0.65
0.35
0.01
18.6 (+8.3)
24.2
Table 7: Score Fusion Parameters. We study different geometric mean fusion parameters by com- paring their AP, LVIS. All results based trained F-VLM R50 model. Default settings are in gray...
(a) VLM-score weights for novel
(b) VLM-score weights for base classes. We fix 0.65 and 0.01.
(c) Temperature for VLM logits. We fix 0.65 and
classes, We fix 0.35 and T
0.01.
a 0.35
3
AP,
AP
a
AP,
AP
AP,
AP
0.0
11.3
22.9
0.15
18.2
0.0025
14.9
18.7
0.45
17.8
24.2
0.25
18.8
24.5
0.005
17.3
22.2
0.65
18.9
24.2
0.35
18.9
24.2
0.01
18.9
24.2
0.85
16.1
22.5
0.55
18.7
22,4
0.02
17.8
24.8
1.00
10.3
17.8
0.75
17.3
17.8
0.04
13.9
Table shows that increased feature pyramid capacity improves the base categories significantly (APc. APy) without compromising the novel categories (AP,). Although it is common to improve the base categories at the cost of novel categories, enlarged feature pyramid leads to improvements on all categories including a slight improvement of 0.1 on AP,.
Table 8: Feature Pyramid Capacity. We observe that larger feature pyramid improves the base categories (APe, APy) without compromising the novel categories (AP,).
Backbone
Feature Pyramid
APr
AP.
AP,
AP
R50x64
FPN (Lin et al., 2017)
32.8
35.4
35.4
34.9
R50x64
FPN (N 12)
32.9 (+0.1)
37.5 (+1.9)
38.3 (+2.9)
37.0 (+2.1)
A.4 BACKGROUND WEIGHT
We study the effects of background weight (Zareian al., 2021; Zhong al., 2022) F-VLM. Consistent with the findings in (Zhong et al., 2022), we found that a background weight of 0.9 is slightly better than the default 1.0 Table Therefore, use background weight 0.9 default. All results are average over 3 independent runs.
B COMPUTATION-PRIEDDI TRAINING
To facilitate comparison with the broader research community, we validate the efficacy of F-VLM in more computation friennlly 1 x (12 epochs) and 3x (36 epochs) settings (Wu et al., 2019) by using smaller batch size and no large-scae-jittering (LSJ) augmentation (Ghiasi et al., 2021). The results listed in Table 10.
We observe that F-VLM is robust to the number training epochs, batch size, with/without LSJ (Ghiasi et al., 2021) for both the smallest and largest backbones. This stands in contrast to the sensitivity of fully supervised learning to these syper-parametent and is consistent with our findings in Table that frozen backbone contributes the training efficiency and stability.
15

---

Published as a conference paper at ICLR 2023
Table 9: Background Weight. We study the effects of background weight of F-VLM LVIS.
Backbone
Background Weight
AP,
R50
1.0
18.3
R50
0.9
18.4 (+0.1)
R50x64
1.0
32.4
R50x64
0.9
32.9 (+0.5)
Table 10: Benchmark computation-friendl training recipes. By leveraging frozen backbone, F-VLM is robust to shorter schedule and smaller batch size. All results are reported as the average over 5 runs. Our default settings in gray.
Backbone
LSJ
# Epochs
Batch Size
AP,
R50
12 (1x)
16
18.1
R50
36 (3x)
64
18.5
R50
100
256
18.6
R50x64
12 (1x)
16
31.9
R50x64
36 (3x)
64
32.6
R50x64
100
256
32.8
C EXPLORING THE STRUCTURE OF FROZEN FEATURES
To understand the effectiveness of F-VLM, we perform k-means clustering probe the structures present in the frozen VLM features (e.g. CLIP). We use CLIP R50x4 backbone and LVIS dataset for visualization. Only the last layer output features are used for clustering, because these features can be used for zero-shot region classification at the same time. Figure 5 demonstrates that the features form nice clusters around salient objects of the scenes (e.g., skis, motorbikes, people), and naturally separate object parts (e.g., donut toppings, bus wheels) without explicit supervision. We believe these emergent properties of frozen VLM are promising avenues to push open-vocabulary detection beyond the domains of existing detection datasets.
D ANALYSIS OF TRANSFER DETECTION BENCHMARK
Many existing works benchmark transfer detection across common detection datasets (Gu et al., 2022: Du al., 2022: Li et al., 2022b; Minderer et al., 2022). In particular, the LVIS to COCO and Objects365 transfer detection a recent benchmark proposed by Gu et al. (2022). We analyze and report the vocabulary overlap percentage between COCO/Objects365 and LVIS base categories in Table 11.
Table 11:
Transfer Detection Vocabulary Overlap. We observe substantial overlap between COCO and Objects365 vocabulary and LVIS base categories.
Method
COCO
Objects365
Name matching
85%
56%
Name matching + near duplicate removal
91%
63%
Simple name matching shows clear overlap, while the removal of near duplicates (e.g. synonyms and non-alphabetic character removal) reveals even more. addition, we notice that COCO has more overlap than Objects365 due to its smaller vocabulary. These results show the limitation of existing transfer detection setup, and we encourage the community to move towards larger transfer detection benchmarks with less vocabulary overlap.
16

---

Published as a conference paper at ICLR 2023
Image
K=3
K=6
Image
K=3
K=6
Figure 5: Understanding the frozen VLM feature clusters. Salient objects and object parts emerge naturally from the clustering frozen VLM features.
E MEMORY USE
The memory consumption of F-VLM is almost the same Mask R-CNN, with all class-specifes heads changed to class-agnostic. F-VLM uses the same batch size as Gu et al. (2022) the default settings, although we found it works well even with much smaller batch size and epoch length (see Appendix B).
Moreover, F-VLM has significant memory saving potential compared to existing approaches that fine-tune the backbone, especially with large backbones. At training time, F-VLM does not need to store forward-pass activations, gradients gradient moments, and the memory use of the backbone is just the backbone weights plus a small amount of current activations. This makes F-VLM highly memory efficient especially with large backbones. In practice, the actual memory depends on the low-level implementation of each deep learning library. In pytorch, for example, "torch.no "raad())"rdd( context manager 2 can enable such behavior.
F CAN WE USE OTHER VLMS?
In this work, we adopt the widely used CLIP (Radford et al., 2021) develop a simple open- vocabulary detection recipe based frozen backbones. Moving forward, we believe it would be very interesting to explore different pretrained VLMs, which may involve substantial modifications to F-VLM. For example, ViT-based pretrained VLMs (Radford al., 2021; Li al., 2021; 2022a; Zhai et al., 2022) have become popular recently. These VLMs require single-scale ViT-based detec- tor such ViTDet (Li et al., 2022c) adapt them for open- vocabulayy detection. To use VLMs like important to consider how to efficiently
ALBEF (Li et al., 2021) and BLIP (Li et al., 2022a), compute all-pair region-text similarities with multimodal encoders (as opposed to dual encoders). Furthermore, it is an open question how to use captioning VLMs (Wang et al., 2021: I; Hu et al., 2022)
http://yywoochhnnn//osssshheeeee grad.
17

---

Published as a conference paper at ICLR 2023
masked multimodal VLMs (Singh al., 2022) for voca-vocyy detection. We believe these are all interesting directions for the community to explore.
G IMPLEMENTATION DETAILS
Table 12 summarizes the eyper-parametery we use for LVIS and COCO experiments. On LVIS, we adopt the same frozen backbone) and a background weight following Zareian et al. (2021); Zhong et al. (2022) (see A.4). The hyper-parameter differences COCO are to mitigate overfitting to the ZSD-COCO split (Bansal et al., 2018) 48 categories, which significantly smaller than the 800 LVIS base cat- egories. This is necessary because F-VLM does not use other objectives e.g. knowledge distillation or weak supervision to counter-balance overfitting.
as Gu et al. (2022) except for a shorter schedule (due to
Table 12: F-VLM hyper-parameter configuration.
Configuration
LVIS
COCO
Optimizer
SGD
SGD
Momentum
8 0.9
= 0.9
Weight decay
le-4
le-2
Gradient Clipping
none
none
Learning rate (LR)
0.36
0.02
Step decay factor
0.1x
0.1 [0.9, 0.95, 0.975]
Step decay schedule
[0.8, 0.9, 0.95]
Warmup LR steps
3.2c-3 lk
3.2e-3 / lk 11.25k
Total Steps
46.1k
Batch size
256
64
Epochs
118
Augmentation
LSJ (Ghiasi et al., 2021)
LSJ (Ghiasi et al., 2021)
NMS Threshold
0.5
0.4
Base VLM weight
0.35
0.2
Novel VLM weight 8
0.65
0.45
Background weight
0.9
0.2
VLM Temperature T
0.01
0.01
We observe some differences in the optimal With the R50x64 backbone, we notice improvement of 1.0 AP, when we use T'' 0.02 as opposed to the default T 0.01. For R50 backbone, we notice an improvement of 0.5 AP, when we apply gradient clipping of 1.0 maximum gradient norm as opposed to none. We report the performance using the optimal settings for these two backbones.
for different backbone architectures.
H VISUALIZATION
We visualize more F- VLM outputs on LVIS novel categories and transfer detection to Objects365 in Figure 6. On LVIS, F-VLM is able correctly detect many rare categories including baguet, neck- erchief, tabasco sauce, and gourd. On Objects365, F- F-VLM can detect many categories in complex scenes including ducks, traffic sign, street light, and air conditioner. These confirm that approach is effective for novel category detection and transfer detection to another dataset. We use the R50x4 backbone for this visualization. The model was trained on the LVIS base categories following the main benchmark of the paper.
I APPLICATION ON EGO-CENTRIC DATA
key benefit of open-vocabulary detection to test on out-of-distributin data with categories given by users the fly. Thus, apply F-VLM Ego4D (Grauman et al., 2022). real-world ego-centric application. We train V-L on mixture of full LVIS, Objects365, and COCO datasets expand its training vocabulary for application in the wild, and use the R50x 16 backbone for this
18

---

Published as a conference paper at ICLR 2023
BB
OWN
(a) LVIS novel category detection. F-VLM can detect many novel categories despite its simplicity using frozen VLM. The white arrows point the novel objects correctly detected by F-VLM. For clarity, only show the novel categories.
(b) Objects365 transfer object defection. F-VLM can be applied to a new dataset and detect many challenging categories without further finetuning.
Figure 6: F-VLM Visualization on LVIS novel categories and Objects365 transfer object detection.
experiment. The model is not trained on Ego4D in order to evaluate for transfer detection. The categories are provided by the user based on visual inspection of the video.
For the indoor scene, the category names provided by the user are as follows: plate, cabinet, stove, towel, cleaning rag, ventilator, knob, sauce and seasoning, steel lid, window, window blinds, plant, light switch, light, door. carpet, exit sign, doormat, hair, door lock, tree, poster the wall, sticker on the wall, faucet, recycle bin, rack. hand, can, carton, trash, Christmas tree, plastic container, fridge.
19

---

Published as a conference paper at ICLR 2023
Image
F-VLM
Image
F-VLM
(a) Indoor scene. Under the challenging viewing angles, occlusion, and lighting conditions, F-VLM still manages detect many objects in the scene.
Image
F-VLM
Image
F-VLM
$5
(b) Grocery store scene. The scene is very crowded with a wide variety of objects. F-VLM is able to detect many of them.
Figure 7: F-VLM Visualization on Ego4D (Grauman et al., 2022) transfer object detection. Novel categories detected: light switch, light, door lock, sauce and seasoning, bag of candies, canned food, and burrito.
For the grocery store scene, the category names provided by the user are as follows: exit sign, poster, chocolate bar, bag of candy, bag of cookies, snack, oreo, soy sauce, apple, pear, orange, grapes, price tag, cereal, instant noodle/ramen, cracker, ATM machine, instant noodle, wooden basket, red ramen bowls, magazine, drugs and medicine, Mayo, Ketchup, Cup noodle, burrito, Lays/Sun chips, seasoning sauce, black carton, salad dressing, canned food.
Figure 7 shows that F-VLM is able detect many objects in the ego-centric videos despite the large domain shift and challenging viewing conditions. In particular, is able to detect many novel categories present in the training set, such light switch, light, door lock, sauce and seasoning. bag of candies, canned food, and burrito.
20


Reviews:
[{"review_id": "aeO1sJWRxd", "reviewer": "Reviewer_e3Lv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is about open-vocabulary object detection (OVD), where detection models are trained from a set of base categories with bounding box annotations, as well as a set of image-caption pairs. Like prior works, this paper leverages existing, pre-trained vision & language models (VLM) like CLIP that were trained on large quantities of image-caption pairs. Unlike prior works, this work shows that an object detector can be put on top of the frozen VLM. Specifically, the proposed object detector consists of the frozen VLM as the backbone, while detector-specific layers (region proposals, feature pyramid network, and region classification) are trainable. The final classification of each region is a combination (geometric mean) of detector scores and off-the-shelf VLM scores. The paper demonstrates that state-of-the-art results can be achieved with a frozen VLM, while the training compute requirements can be reduced significantly.", "summary_of_the_review": "Overall, I think the paper should be accepted. Although the technical contribution is simple, the proposed framework is effective and gives state-of-the-art results. The paper is well written and the authors plan to release code.", "strength_and_weaknesses": "Strengths:\n- The proposed approach is simple, but highly effective considering the reduced training compute requirements. The paper shows a novel way how to leverage pre-trained VLMs for OVD. I think the reduced compute requirements are interesting to the research community, particularly for institutions with limited compute budgets.\n- The comparison with prior works is good; using standard benchmarks in a fair setting.\n- I think the ablation studies are good; I like the summary of findings where details and tables are put into the appendix.\n\nWeaknesses:\n- Given that the VLM is frozen, I think the paper would benefit from evaluating different choices of the VLM, beyond the for ResNet-variants in CLIP. For instance, ALBEF [A] or BLIP [B] are publicly available.\n- Table 4 and the corresponding analysis about the training compute requirements is great, but I would find it much better to make this analysis more prominent in the paper (instead of the last item after ablation studies). After all, this is one of the key arguments this paper tries to convey: keeping the VLM frozen, you only need to train detector-specific parameters and still get good results with significantly reduced training compute. While I expect future methods to achieve better results by training the whole model (maybe better finetuning and directly including image-caption data), the training efficiency of this work would still stand.\n- A recent, but concurrent paper from ECCV [C] could be included in the discussion, because it also uses a \"frozen\" VLM, but for creating pseudo labels (the \"fast\" version also applies ROI-align on CLIP-image-encoder features).\n- I understand that the cross-dataset experiment (LVIS to COCO and Objects-365) has been used in prior work, but I do not think this is a great experimental setup to claim any generalization. As mentioned in the paper, the overlap of categories is high ... also evident from the chosen hyper-parameters (beta = 0). First, I would suggest to explicitly state the amount of overlap of the label spaces to make this point more clear. Second, I think it is better to move this experiment into the appendix (the reader can then still see the comparison with prior work), and instead propose novel ways to demonstrate generalization. I like the evaluation on Ego4D. Is a quantitative evaluation possible? An alternative could be the ODinW benchmark [D]. Although the localization ability is not evaluated, one could try to use ImageNet for such label space generalization experiments, with the assumption that the highest scoring bounding box per image is chosen as the prediction.\n- What about the memory consumption for the proposed model? Table 3 does not mention this aspect. Related to that, was the batch size increased compared to the ViLD training settings?\n\nReferences: \n* [A] Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. Li et al. NeurIPS 2021\n* [B] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. Li et al. ICML 2022\n* [C] Exploiting Unlabeled Data with Vision and Language Models for Object Detection. Zhao et al. ECCV 2022\n* [D] ODinW benchmark https://eval.ai/web/challenges/challenge-page/1839/overview", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TnR8Vvi6ByM", "reviewer": "Reviewer_kxyx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use a frozen vision and language pre-trained model (CLIP) for open-vocabulary object detection. The image encoder from CLIP is used as the backbone and the textual encoder from CLIP is used for region proposal classification. A trainable detector head is added to generate region proposals (for localization), which are then classified into a set of base and novel categories.", "summary_of_the_review": "Overall, this paper provides an appealing approach for open-vocabulary object detection. I'll update my rating according to other reviewers' comments and the authors' responses.\n\nI'd like to encourage the authors to further address the following items (they are not the weaknesses, or at least not unique in this paper, so I didn't list them above).\n\n1. According to Eq.(3), the RoIAlign operator is used in both the training and inference phases. Adding the illustration of RoIAlign in Fig. 2(a) will probably make it clearer.\n\n2. For comparisons with different models, especially the system-level comparison, it may be a good idea to list the number of parameters of different models. So we can clearly see the relationship of the models' capacity and their accuracy.\n\n3. The qualitative results of cross-dataset generalization is great. But the quantitative evaluation is not a \"in-the-wild\" setting. Basically, the region classification is dependent on the cosine similarities of a region's feature w.r.t. to a set of candidate categories. What if the vocabulary of the novel categories is very large with a lot of class names that are not available in an image? Would the accuracy of the model significantly decrease as more and more such distracting categories added? It would be great to see such comparisons (it does not require re-training the model).\n\n4. Only certain values of $\\alpha$ and $\\beta$ values are studied in Table 6. Is it possible to report more ablation results and visualize them using plots?", "strength_and_weaknesses": "Strengths:\n1. The overall framework is conceptually simple yet effective. Directly using a frozen pre-trained vision and language model is easier to deal with than performing knowledge distillation. And also the training cost is significantly lower than others because the entire CLIP model is frozen. It is inspiring for future work.\n\n2. Strong quantitative results are reported on LVIS and COCO. Qualitative cross-dataset generalization results on Ego4D are also shown.\n\nWeaknesses:\n1. There are four hyper parameters to tune in the proposed approach: $\\tau$ in Eq.(2), $T$ in Eq.(4), and $\\alpha$ and $\\beta$ in Eq.(5). They can be tuned on a validation set. But in real-world use, it is hard to carefully tune them all.\n\n2. The paper claims that a bronze VLM \"retains the locality-sensitive features necessary for detection\". But only qualitative results are shown in Fig. 1 and 4. The paper implicitly assumes that the localization module (regional proposal generation) can generalize well from base categories to novel ones with quantitative verification. To validate this assumption, we can check the recall of the model on both base and novel categories. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7fJBc6HizFg", "reviewer": "Reviewer_x64o", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents F-VLM, which tackles open-vocabulary detection based on frozen pre-trained vision-language models (VLMs). The authors observe that frozen VLMs retain locality-sensitive features and are strong region classifiers. By only training a detection head upon frozen VLMs and utilizing regional VLM features for open-vocabulary classification at inference time, the F-VLM achieves SOTA on LVIS benchmark as well as competitive results on COCO benchmark and cross-dataset transferring. In addition, F-VLM shows compelling scalability and achieves significant training speed-up and computing saving. ", "summary_of_the_review": "Overall, the paper explores a new direction that uses frozen vision-language models for detection and provides useful experimental results which reveal the potential and scalability of this direction. The unfair comparison might be an issue and it could be better if the paper modulates its claims.", "strength_and_weaknesses": "## Strengths:\n1. The paper is well-written and easy to follow \n2. The proposed method is simple, fast and computing-saving for training, free-from knowledge distillation and weakly supervised learning,  while effectively exploiting the VLMs’ ability for open-vocabulary region classification. \n3. The paper provides detailed ablative studies on the F-VLM\n\n## Weakness:\n1. The performance of the proposed method is inferior to existing methods under fair comparison. E.g. “R50 Comparison” in Table 1, Table 2, and “F-VLM-R50 (Ours)” in Table 3. Propmt optimization and SoCo pre-training are not orthogonal to F-VLM because it seems that these two strategies seem to be not directly applicable to F-VLM (F-VLM uses fronzon VL models), while DetPro can scale its methods to use ViT-L or R50x64. The paper also adopts heavier data augmentation and longer training schedule. \n2. The insights are limited. Basically, the paper only finds a way and tells users that using a frozen VL model in the detector could achieve compatible results with scalability. But the paper could discuss more the effects of VL models, especially those of different publically available pre-trained VL models.\n\n## Questions:\n1. The training recipe (Implementation Details at Sec 4, page 6) of the detectors in the paper is not affordable and thus not widely adopted in the community (large-scale jittering, image size 1024x1024 and batch size 256) and might be unfair for comparison. What about training on a regular 12-epoch or 36-epoch schedule with batch size 16?  \n2. The ablation study on score fusion in Sec 4.3. What about classifying regions solely on the RoiAligned features or solely on the embeddings from the detector head? \n3. In the 6th row of the 2nd paragraph of Sec 3.4, what’s the meaning of “but train the detector head with $R(\\cdot)$ online in a single stage”? From what the context described, the RoiAlign $R(\\cdot)$ mentioned here is only used at inference time.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster