Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition
Authors: Shuhuai Ren
Aston Zhang
Yi Zhu
Shuai Zhang
Shuai Zheng
Mu Li
Alex Smola
Xu Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: Prompt Pre-Training
CLIP
Open-Vocabulary Visual Recognition

Abstract: This work proposes POMP, a prompt pre-training method for vision-language models. Being memory and computation efficient, POMP enables the learned prompt to condense semantic information for a rich set of visual concepts with over twenty-thousand classes. Once pre-trained, the prompt with a strong transferable ability can be directly plugged into a variety of visual recognition tasks including image classification, semantic segmentation, and object detection, to boost recognition performances in a zero-shot manner. Empirical evaluation shows that POMP achieves state-of-the-art performances on 21 datasets, e.g., 67.0% average accuracy on 10 classification datasets (+3.1% compared to CoOp) and 84.4 hIoU on open-vocabulary Pascal VOC segmentation (+6.9 compared to ZSSeg).

Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition
Shuhuai Ren Aston Zhang Yi Zhu', Shuai Zhang', Shuai Zheng',
Mu Lit, Alex Smola', Xu Sun
National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
Amazon Web Services
Abstract
This work proposes POMP, a prompt pre-training method for vision-language models. Being memory and computation efficient, POMP enables the learned prompt to condense semantic information for rich set of visual concepts with over twenty-thousand classes. Once pre-trained, the prompt with a strong transferable ability can be directly plugged into variety of visual recognition tasks including image classification, semantic segmentation, and object detection, to boost recogni- tion performances a zero-shot manner. Empirical evaluation shows that POMP achieves wate-offthe-ee performances on 21 datasets, e.g., 67 67.0% average accuracy on 10 classification datasets (+3.1% compared to CoOp) and 84.4 hloU on open- vocabulary Pascal VOC segmentation (+6.9 compared to ZSSeg). Our code is available at at / com/ amazon- science/promp perttrarrttrnnnng
Introduction
has been a new norm to formulate visual recognition tasks (e.g., image classification, object detection, and semantic segmentation) as eaguage-guidee visual recognition or vision-and dision-and-language problems [42 19 61). language-guided visual recognition, categories of images are represented by natural language rather than discrete label IDs, and the semantics between images and their corresponding textual descriptions are often aligned via a constrastive loss during training 42. Model inference also becomes an image-to-fest matching problem, where text prompts like "a photo of [CLASSNAME]" are curated as text descriptions of images. By varying the [CLASSNAME] placeholder and computing the similarity between text descriptions and images, class name and consider it as the predicted target class. A significant benefit of this language language-guided paradigm is that supports open-vocabulary inference, that is, zero-shot recognition for arbitrary categories that may not even have been seen during training, thanks to the flexibility modifying the class names in the textual prompt 45
can identify the most suitable
As the context for class names, the text prompt plays a critical role in language-guided visual recognition models. A good prompt should holistically express the semantics of visual categories to better elicit knowledge learned by languae models (VLMs) during pre-training. There are two popular types of prompts: hard prompts (e.g., photo of a [CLASSNAME]) and soft prompts. Soft prompts are learnable token embeddings that can be fine-tuned given some input data, and have been demonstrated to be more effective and stable on downstream tasks than hard prompts [65] 35 63] However, traditional prompt tuning methods usually fine-tune the soft prompt on task-specific datasets with limited number of class labels, making difficult to generalize novel classes and across tasks 45 2
For example, when transferred between the two downstream datasets of Flowers
44)
*Correspondence to Aston Zhang <az @ astonzhang.comm
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

and DTD [9], the soft prompt fine-tuned on DTD only achieves 33.4% accuracy on Flowers 102, significantly lower than the 61.8% accuracy of hand-crafted prompt on Flowers102, demonstrating severe overfitting issue 49
In this work, aim to learn universal soft prompt that covers a broad set of visual concepts while being task-agnostic. Specifically, we pro- pose PrOMpt Pre-training (POMP), a method for scaling up prompt learning on the ImageNet- 21K dataset, which has over twenty- -thosy-t classes organized by the WordNet (37 hierarchy.
Oroagetice CEN)
Sepmantition
22.6
(Orms-delasee)
51.6
$6.6
52.3
[coCO
aeghnetttiin
218
(Cros-demssh)
The set of classes in ImageNet-22 2 K includes general and long-tail visual categories of various semantic granularities, which have been proven to provide better downstream results for large models [30 [4]. Pre-training on this large-scale dataset helps condense semantic information into the soft prompt for universal visual discrim- ination. Once pre- pre-trained, this universal prompt (i) can be easily applied to downstream datasets to improve model performance in zero-shot set- tings; (ii) is compatible with both region-level and pixel-level visual patterns, making it useful for various vision tasks such object detection and semantic segmentation.
12.8
50.
92.4
14.1
.6.7
Ser oete
$1.8
Object (FI))
PAGCAL VOC)
32.2
Sumontic CDCo Stam
23.6
15.
Delactioe Bans
Object
Previous SOTA
POMP [Ours)
Figure 1: POMP outperforms previous state-of-the- art models on a broad range of visual recognition tasks and datasets.
However, pre-training prompt with such massive class set is challenging due to generally prohibitive computational costs. During prompt pre-training, the activation of the whole text encoder needs to be kept independently for every class and the memory consumption increases proportionally to the number of classes. In short, pre-training prompts on ImageNet-22 requires over 300 GB GPU memory with traditional methods like CoOp [65]. In POMP, solve this issue with simple class sampling strategy, local contrast, which reduces the GPU memory requirement dramatically to less than 16 GB. Moreover, we propose local correction strategy to reduce the bias caused by class sampling and improve the generalization of the pre-traines prompt.
Experimental results in Figure I show that POMP outperforms previous state-of-the-art (SOTA) models on a broad range of visual recognition tasks and datasets. Specifically, compared to zeroshot CLIP 42. POMP improves the accuracy on ImageNet-211 IK by gain of +2.9%. It also achieves an average accuracy of 67.0% when transferred to 10 downstream image classification datasets, which is 3.1% higher than CoOp [65]. For semantic segmentation, POMP achieves 39.1 1 hloU open-vocab COCO Stuff and 84.4 hloU on open open-vocab Pascal VOC, outperforming ZSSeg [61] by +1.3 and +6.9 hloU, respectively. For object detection, POMP achieves 57.9 and 22.9 AP when transferred from LVIS to COCO and Object365, surpassing Detic (67 by +1.9 and +0.8 AP50, respectively.
2 Related Work
2.1 Language-Guided Visual Recognition
Language-guider visual recognition usually leverages VLMs as foundation models. Representative VLMS like CLIIP 42) consist of an image encoder and a text encoder, which are used to encode image-text pairs into a joint feature space for learning the semantic alignment between vision and language 7 [46). After Afterbeing pre-trained on large-scale image-text pairs, CLIP-like models (28 62 33 are able map images to their corresponding language descriptions, allowing visual recognition to generalize in the wild. This language driven modeling paradigm also facilitates other vision tasks, including semantic segmentation 66] 32 B and object detection [19
15 67). These works typically designed a two-stage framework: it first leverages the pre-traine pre-trained proposal network to extract features of specific visual patterns (e.g., segment mask and region) and then conducts classification in the same matching style as CL.PP. The class descriptions for matching are synthesized using prompts, and in this work, we pre-train a soft prompt for VLMs on ImageNet-21 K to further enhance their zero-sho shot generalization ability.

---

2.2
Prompt Tuning
In order to adapt VLMs to downstream tasks, recent research proposed parameter- effficie tuning method named prompt tuning. CoOp [65] first proposed to replace the hand- cand-crafted prompt with learnable vectors (also known as a soft prompt) for fine-tuning while freezing the entire pre-trained parameters. VPT [2], on the contrary, moved the learnable vectors from the text side to the image side, and proposed to concatenate the "visual soft prompt" and the patch sequence of an image as the input for fine- thee-tunigg Prompt tuning was also used in other visual recognition tasks, such as object detection [1], semantic segmentation [43], and video recognition [39]. Over manual prompt engineering, the soft prompt optimized with few-shot data has achieved significant performance improvements, but only fitting one specific downstream dataset.
To enhance the generalization of the soft prompt to a wider range of unseen classes and datasets, CoCoOp (66) modeled the context condition on input images. A recent work MaPLe [29] appended the soft prompt to the hidden representations at each layer in both the text and image encoder. In sharp contrast to previous approaches, we propose to pre-train a universal soft prompt on large-scale datasets with massive visual categories. Such a pre-trained prompt is task-agnostic, allowing for direct transfer to various downstream datasets without fine-tuning.
3 Method
We first review the process of classical prompt tuning for VLMs in $ 3..1 To address the training efficiency issue of previous methods, in $ 3.2 we introduce method of prompt pre-training (POMP) that includes two key components: local contrast and local correction. Our pre-trained prompt can then be transferred to downstream datasets and tasks in zero-shot manner as discussed in $ 3.3
3.1 Preliminaries
Language-guin -uided visual recognition models like CLIP formulate image classification as an image-text matching problem, where the goal is to select the correct textual class name from a predefined class set for the image query. Following 665 29],, we consider CLIP as our vision- Lsionnlanguagg foundation model, for its simplicity in design and wide applicability. CLIP consists of an image encoder Ji and text encoder fr. Given visual recognition dataset ID with a class set of N class names C = [ CLIP manually devises a hard prompt to synthesize textual descriptions f; for each class name c; e.g., t, ="a photo of a [c,]". Then each class description fed into the text encoder to generate the normalized class feature w; = fr (t.)) (ftr E Rd, where d is the dimension of the feature. The concatenation of N class features [Wi,-' ..... wo] Nxd can be considered the class weight of linear classifier for classifying an image. Given an input image v. the image encoder is used to extract (fff)//ll(/fr 22 E Rdd Finally, CLIP calculates the similarity between and
its visual feature all the class features, then predicts the class with the highest similarity as the target class.
To address the inefficient expressiveness of the manual prompt, previous research like CoOp [65] proposed to parameterize the manual prompt as a soft prompt θ, and fine-tune it to fit down- stream datasets. The soft prompt is made up of sequence of learnable token embeddings θ = [0,,0 ...... 0, E R xx where M is hyperparameter specifying the length of the soft prompt and is the dimension of the token embedding. The token embeddings of each class name
e are further appended to the soft prompt t to generate the class feature w; and the prediction probability for the ground-truth class y is denotes as
(9)
exp(x Wy /T)
(9)
( x: θ)
(I)
ΣN exp(x w! (((1/T)
where x w, represents the similarity score and J is a temperature parameter. The parameters of the soft prompt are updated by minimizing the cross-entropy loss
655
((O)=
log P (y x; 0)]
(2)
(x
The gradient of L(O) represented as:
Ve log P (y x: 0) = - Ve(x ((())) + Σ (m Vx [.0)
(3)
3

---

Prompt Pro- Training (Stage 1]
Learnable
Frozen
Bout Brooder
Contrastie Loss
Gilmage
Ennoder
Names
Reatires
Image Feaure
Cosiifcation
$18.4
Zero-Shot Transfer (Stage 2)
Clesdly
Imaag Entoder
GPU Memory
CIS Clases
Image Fecture
Caediration
Promen
Test Encoder
Clissfss
Seggage Proppial
50
per C sasuss
legion Feature
Desection
3.0
GPARID (L0)
PGVC Aroaatt (zom)
CTD (47)
Stanfard Cao (196)
Suncet cam
Imsgelet- acoe
Imageteet (233)
Mook
Networ
SEG
Features
Mask Fertare
Segnentation
Dataset (#class)
Figure Overview of POMP. POMP pre-trains a soft prompt learnable) on the ImgaNet-21K dataset with massive classes, and then directly transfers the learned prompt (@ffrozen) to downstream datasets of image clas- sification (CLS), object detection (DET), and semantic segmentation (SEG) tasks. For DET and SEG, the region and mask proposal networks require pre-training with POMP prompt detection and segmentation source data, respectively (See Appendix[B]
Figure GPU memory overhead (GB) required for prompt tuning on datasets with varying numbers of classes. The memory cost of CoOp on ImageNet-2IK is 316.4 GB, which is generally pro- hibitive. POMP reduces the cost dra- matically to 15.7 7 GB with local contrast among the 1000 sampled classes for op- timization.
It can be decomposed into positive reinforcement for the ground-truth class and negative reinforcement for every class, which are the first and the second terms inside the square brackets of respectively.
Note that both the image encoder and the text encoder are frozen during the prompt tuning process, which allows adapting the soft prompt efficiently to downstream data with very few learnable parameters. Methods along this direction of soft prompt learning include CoCoOp [66] and MaPLe [29]. However, most previous works fine-tune task-specific prompts, which limits their versatility and generalization 49
3.2 POMP: Prompt Pre-Training
Now we present our task-agnostic prompt pre-training method: POMP. Once pre-trained, the learned prompt can be directly used for downstream tasks without fine-tuning (see Figure D As introduced in $ we propose to pre-train the soft prompt on the ImageNet-211 K dataset for universal visual discrimination. Although the prompt tuning methods such as COOP and CoCoOp are parameter- efficient, they still incur computationally prohibitive training costs when applied to large-scale datasets with massive number of classes. Recall that the learnable parameters O are embedded the text input, while the loss is calculated at the output layer of the text encoder. For every class description, we need to allocate nearly 15 MB of GPU memory to preserve the state of the entire frozen encoder (Transformer-basee [53] with 12 layers), and propagate the gradient back through the last layer to the first layer, to update the soft prompt @ Accordingly, the computational and caching cost of prompt tuning is proportional to the number of classes N. As shown in Figure 3. for general large-scale datasets like ImageNet-211 with more than twenty-thousan classes, traditional tuning methods will allocate 21K x55 MB (more than 300 GB) of GPU memory, which is generally prohibitive.
To enable prompt tuning on massive classes and acquire the capability of global visual discrimination, we introduce a saaiiingefffiiien algorithm called POMP, which reduces the GPU memory and training time of prompt tuning dramatically. POMP has two major components: local contrast and local correction. The former decreases the number of classes for contrastive learning through negative class sampling, and the latter reduces the bias caused by local contrast by adjusting the similarity scores of negative classes. We detail these two components in the following.

---

3.2.1 Local Contrast
Diseriminating all classes during contrastive learning is the source of training inefficiency. In order to alleviate this problem, we propose to narrow the scope of contrastive learning from global to local, and only require the model to identify the ground-truth class of the input image from subset of the full class set. The class subset is sampled at each training step, allowing the model to discriminate within an ever-changing set of categories, and gradually restoring the relationship among all categories.
Specifically, given an input image, we sample K classes (K is much smaller than the total number of classes, N), including the ground-trute class and K 1 negative classes. We use straightforward yet effective proposal distribution of uniform distribution for negative class sampling. where every negative class has equal probability, i.e., P = 1 1/(N - 1), of being sampled. We also explore alternative types of proposal distribution, such as frequency- saauueney-b-eedd based and similarity- -baed distributions. However, our experiments reveal that POMP with the simple uniform distribution considers both common and rare classes, as well as easy and difficult classes, resulting to the best performance. Please refer to Appendix E. for details.
After sampling, we denote the set of the negative classes as M,, with IM -. By using the local contrast, we can significantly reduce the training overhead to fraction of K/N of the original one. Upon completion of training, we can use the full class set compute the prediction probability of each image. Overall, the motivation behind our local contrast is analogous to that in the NCE-based contrastive learning frameworks [52 57]. In these frameworks, the contrast is performed by sampling a batch of instances due to computational limitations and computing the loss within the batch as an
empirical estimation for the expected contrastive loss 25 22 ST 68
3.2.2 Local Correction
Given that the local contrast component necessitates reduced number of negative classes, the negative reinforcement in the vanilla gradient in (3) is diminished to K/N. As a result, the prompt optimization direction is inevitably biased due to the absence of other negative classes. To mitigate this bias and enhance the model performance, we add a local correction term m to the logits of the sampled negative classes w /T (i + y). which serves as margin 160 45 69 between the
positive and the negative logits. Accordingly, the final prediction probability of POMP is denoted as:
exp(x wy (/007)
Ply x;
0)
exp(x T wy
(9) ( /T) Σ exp(x T w;
/ + m)
The local correction term m encourages the positive logit to be larger than the negative logits by a certain margin, resulting in a more stringent decision boundary:
C+: :
x /T ++m, m, i i
Therefore, compared to the prediction probability without local correction, 4 makes the decision boundary more robust against the unsampled negative classes and enforces the learning of more discriminative class features [60]. This will improve model regularization and robustness across datasets and domains (to be shown in 4.4. Different from other margin- based losses that use fixed margin [T] [4],, our margin m is designed to adaptively adjust itself based on the value of K:
m - log ((K- - 1) - -1)).
(5)
is worth noting that m in 55) is positive scalar. When K N, all classes are included during optimization, and m equals zero. In this case, 4 degenerates to the standard prediction probability in ( As the value of K decreases and the number of visible classes is reduced, the margin m increases to create space for potential class features in the representation space. Our adaptive margin outperforms the fixed margins (which are specified as syper-parameters) for various K, allowing models to maintain optimal performance under different computing budgets (to be shown in $ 4.4).
3.3 Zero-Shot Transfer Learning
As shown in Figure 2 after pre-training, our POMP prompt can be used to synthesize class features for classification with an arbitrary class set, supporting zero-shot inference on downstream datasets
5

---

and tasks. In order to plug the POMP prompt into other visual tasks like semantic segmentation and object detection, we adopt a two-stage framework. In stage one, we use a pre-trained proposal network to generate set of mask or region proposals. In stage two, we classify each proposal with the class features generated by our POMP prompt. Experiments in $ $4.3 will show that the POMP prompt can handle both pixel-level and region-level visual patterns, leading to improved performance in segmentation and detection tasks.
4 Experiments
4.1 POMP Prompt Pre-Training
We take CLIP (ViT/B-16) [42] as the backbone and conduct prompt pre-training on the ImageNet-21K dataset. The number of training samples for each class is 16 (16 shots), and the prompt length is 16. We sample 1,000 classes at each training step, i.c., K 1000 in See Appendix A for details.
4.2 End Task Setups and Implementation Details
To evaluate the generalization of the pre-trained prompt, we directly transfer it to downstream tasks and datasets. Appendix Cists the details of all the datasets. We follow previous works (58 61 19 67 to designate data belonging to two class sets as source data and target data, respectively. The proposal networks are pre-trained on the source data with the source class set, while conducting zero-shot evaluation the target data with the target class set. There are two protocols for the source-target data split. The first is the gpen-vocabulary protocol, where the class set of one dataset divided into two disjoint groups for the source and target data. The second protocol is the cross-dataset protocol, in which the source and target data are from two independent datasets with potentially overlapping class sets. See Appendix B for implementation details.
4.3 Results and Analysis
4.3.1 Prompt Pre-Training on ImageNet-21K
Table[]shows the results of POMP prompt on the ImageNet-21K K test set after pre-training. The traditional prompt learning methods (c.g., CoOp and MaPLe) are trained the ImageNet-Ies IK dataset due to their prohibitive computational cost if trained on ImageNet-2IK (more than 300 GB of GPU memory). On the contrary. our POMP prompt. pre-trained on ImageNet-211 K using less than 16 GB of GPU memory, achieves the highest accuracy of 25.3% based on the CLIP (ViT-B/16) backbone, which surpasses Zeroshot- CLIP by 3.5% and Linear Probe by 4.4%. The VPT method, which uses visual prompts on the image side, does not require training overhead proportional to the number of classes, making applicable to the ImageNet-21 dataset. VPT prepends independent learnable vectors to the hidden states of each layer in the visual back- bone, surpassing linear probing and the previous prompt tuning methods. However, its performance based on ViT-B/16 is still 0.5% worse than ours, demonstrating that our POMP prompt can better distinguish large number of general visual categories. In addition, our method is agnostic to the backbone architectures like ResNet and ViT, and the improvement is consistent.
Table 1: Performance on the ImageNet-21K test set. ZeroshotCLIP and Prompt Ensemble in the top block conduct zero-shot inference. CoOp and MaPLe, indicated in gray in the middle block, are trained on the ImageNet-IK IK dataset due to prohibitive GPU memory consumption trained on ImageNet-21 K. The remaining methods in the bottom block are trained on ImageNet-21K:
Method
ResNer50
VIT-B/32
ViT-B/16
ZeroshorCLIP
17.5
19.8
21.8
Prompt Ensemble
42
18.8
20.9
23.5
CoOp
16.6
18.1
20.8
MaPLe 29
21.6
24.2
Lincar Probing
6.5
18.2
20.9
VPT 12
21.8
24.8
POMP (Ours)
20.2
22.2
25.3
Cross-dataset and Cross-domain Image Classification. Our POMP prompt, which has been pre-trained on large number of classes, demonstrates strong generalization ability. As shown in Table POMP achieves the highest average accuracy of 67.0% when transferred to downstream image classification datasets, outperforming CoOp by 3.1% % and surpassing the previous SOTA in

---

Table 2: Cross-dataset and cross-domain evaluation for image classification. The backbone is
ViT/B-16. Overall, POMP achieves the highest average accuracy, indicating better generalization.
Target datuse-)
Target (cruss- domai-)
Catecht
OxtondRet
SaanfrrCaars
Flaven
Foad11
Arcrot
SUN397
EwoSAr
UCPIO
Anorage
InV2
BongeMootd
Ananage
DMD
prompt
93.3
88.2
67.4
23.7
44.3
42.0
65.1
63.7
60.9
CoOp 65
93.7
89.1
68.7
18.5
64.2
41.9
46.4
63.9
64.2
48.0
49.7
75.2
59.3
CaCoOp 66
94.4
90.1
65.3
71.9
86.1
22.9
67.4
45.7
45.4
68.2
68.7
64.1
48.8
50.6
76.2
59.9
LASP 1
89.4
93.7
70.9
24.9
46.
45.9
660
49.2
51.3
McPLe 29
93.5
90.5
65.6
72.2
86.2
21.7
67.0
46.5
48.1
68.7
66.3
61.1
19.2
50.9
77.0
60.3
POMP (Ours)
95.0
89,5
66.8
72.4
86.3
25.6
67.7
46.3
52.1
68.5
67.0
63.8
49.8
51.6
77.9
60.8
7/10 datasets. This is due to the fact that, after learning with enormous long-tail categories, POMP can provide a more expressive context for fine- ane-ggained visual concepts such as specific objects and scenes, resulting in improved performance (+0.9%), as well as SUN397 (+0.7%) and EuroSAT (+4%). Furthermore, POMP is more robust to domain shift and achieves new SOTA with 60.8% accuracy out-of-domain variants of the ImageNet dataset.
datasets like StanfordCars (+1.2%) and Aircraft
Training Efficiency. POMP also achieves comparable accuracy to the classical prompt tuning methods when fine-tuning on specific downstream datasets, but significantly reduces the training cost. Table shoos the performance of prompt tuning on ImageNet-
Table 3: Prompt tuning ImageNet-IK: POMP
(K = 128) achieves comparable accuracy with CoOp and CoCoOp. but using less than 19% GPU memory and 50% training time.
using vi- sual backbone of ViT-B/16 and 16 shots. The epoch 50 for CoOp and POMP, and 10 for CoCoOp. CoOp generates all the 1000 class features at each training step, which consumes 28 GB of memory and takes 5.9 hours to finish the fine-tuning. The training time of CoCoOp is longer because it devises instance- specifcc prompts that require independent forward pass for each image. Compared to these baselines, POMP (K = 128) achieves competitive accuracy on ImageNet- IK while using less than 19% of GPU memory and 50% of training time, demonstrating its superiority.
Method
Acc. (%)
GPU Mom. (GB)
Training Time (h)
CoOp
28.2
CoCoOp
70.1
28.3
27.5
128)
5.3
POMP (K 256)
71.4
8.8
3.3
POMP (K- 512)
71.6
15.9
4.2
4.3.2 Open-Vocabulary Semantic Segmentation
Table c shows the results of our method on open-vocabulary COCO Stuff and Pascal VOC. POMP outpertorms the previous state-of-the-arr method, ZSSeg (6]) with higher hloU of 39.1 and mloU-unseen of 38.2 on COCO Stuff. On Paseal VOC, the improvement of POMP more significant with +6.9 hloU and +4.3 mloU-unseen. Figure Fillustrates qualitative results open-vocabulary COCO-Stuff, where POMP demonstrates a stronger ability to distinguish background categories compared to ZSSeg. For example, in case (1), ZSSeg misclassifies the classes of playingfield as dirt, while the POMP prompt with richer contextual semantics better expresses the difference between regular land and the playingfield with specific textures, thus facilitating the matching of the visual region with the ground- truth class.
POMP also demonstrates its generalization ability in cross-dataset settings. Taking standard COCO Stuff as the source dataset for mask proposal network pre-training, POMP achieves 20.7 mloU and 51.1 mIoU when transferred to the target datasets of ADE20K and PASCAL Context, respectively. outperforming ZSSeg by +1.3 mIoU and +0.3 mloU. Overall, POMP obtains remarkable gains over previous works in all settings.
4.3.3 Open-Vocabulary Object Detection

---

Table 4: Comparison with state-of-the-art meth- ods on COCO Stuff dataset and Pascal VOC dataset. POMP and ZSSeg share the same mask proposal network and training strategy.
255eg (Baseline)
Open-Vreah COCO Stuff
Open-Vocah Pascal VOC
Mathod
hloU
hioU
mloU
seen
unseen
seen
unseen
POMP
SPNel
58
16.8
20.5
14.3
21.8
73.3
15.0
(Dun)
4
15.0
34.7
17.3
17.7
CaONe
20
18.2
35.5
12.2
39.7
78.4
25.6
Zeg Former
13
34.8
36.6
33.2
73.3
86.4
63.6
25Seg
61
37.8
39.3
36.3
77.5
83.5
72.5
(Ours)
39,9
93.6
76.8
Ground
-buth
Table 5: Cross-dataset evaluation for semantic segmentation. The mask proposal network is pre-trained on standard COCO Stuff.
Figure 4: Qualitative results open-vocabulary COCO-Stuff. Compared to ZSSeg, POMP cor- rectly identifies the background category of playingfield (left) and river (right).
Sanro Dhraett
Theys Dasse
Target Dissset
misU
twisU
Patsll
pACt:
mist
fwinl
PALI
40.8
18.T
600
61
Table 6: Cross-dataset evaluation for object detection. The region proposal network is are-trained on standard LVIS. POMP and Detic share the same region proposal network and training strategy.
Method
Source Datsser:
Standard LVIS
Target Datsset: COCO
Target Dataset: Chjeats365
AP
AP
AP
AP,
AP.,
AP:
AP
AP
AP
AP,
AP..
AP:
AP
APw:
AP
AP,
AP..
AP)
VILD
10
27.5
41.8
29.3
20.6
35.9
43.4
34.1
52.3
36.5
21.6
38.9
46.1
11.5
17.8
12.3
4,2
11.1
17.8
DeiPro
28.4
44.1
Detie
36.8
50.7
38.6
26
46.7
51.7
38.8
56.0
41.9
25.6
42.2
50.0
15.6
22.1
168
6.1
15.6
23.8
POMP (Ouns)
37,2
51.1
39.3
26.5
47.2
52.6
40.3
57.9
43.6
28.1
43.9
50.6
16.1
22.9
17.3
6.2
16.3
24.7
We compare POMP with state-of-the-art meth- ods the open-vocabulary LVIS benchmarks and report results in Table 7 POMP achieves AP, of 26.8 for object detection and 25.2 for instance segmentation. See Appendix D for qualitative results. Under the cross-dataset set- ting, we pre-train the visual backbone on the source dataset of standard LVIS, and evaluate the recognition ability on COCO and Object365. As shown in Table 6 compared to Detic, POMP provides a gain of 1.9 APso on COCO and 0.8 APs0 on Object365, respectively.
Table 7: Comparison with previous SOTA LVIS dataset. AP, is the main evaluation metric for open-vocabulary object detection.
Method
Detection
Instance segmentation
AP:
AP.
AP;
AP
AP.
AP.
AP/
AP
VILD [0
16.7
26.5
34.2
27.8
16.6
24.6
30.3
25.5
DeiPro
32.4
28.4
19.8
28.9
ProngeDee
21.4
23.3
20.3
Detic 67
26.7
36.4
40.3
36.3
21.9
32.5
35.6
32.4
POMP (Churs)
26.8
36.4
40.4
36.2
25.2
33.0
35.6
32.7
4.4 Ablation Study
We decouple the two components of local con- trast and local correction in POMP, and conduct an ablation study to examine their individual contributions. Since removing the local contrast component will lead to prohibitive training cost, we investigate the impact of this component by varying the number of sampled classes K. As shown in Table g the performance of POMP
Table 8: Ablation on the local contrast and local correction in POMP based on the CLIP (1)
Cross-dataset (10 Avg.)
Cross-domaia Avg)
Method
ImageNet-21K
POMP (K 100)
24.1
65.5
59,5
500)
24.9
66.5
60.0
POMP (K- 1000)
25,3
67.0
60.8
improves as K increases. As discussed in $4.3.1
correction
25.0(-0.3)
(-.2()
(9.8
and 3 the local contrast component balances accuracy and cost by adjusting K.
On the other hand, as shown in Table 8 removing the local correction component from POMP
(K 1000) results in decline of 1.2 and 1.0 in the average accuracy of cross-dataset and cross-domain transfer, respectively. This indicates that local correction significantly improves the
8

---

-1.05
27.1
-1.0
1.20
losil correction
-.25
(a) Aireraft.
(b) UCFI01
Figure 5: aaiggn and Cuitform of POMP. For both measures, lower numbers are better. The color of circles and the numbers in the boxes denote the average cross-dataset accuracy over 10 datasets (higher is better).
Figure 6: Projection of image features (points), class features of POMP (intersections of solid lines and sphere) and class features of CoOp (intersections of light dash-dot lines and sphere). Each color represents a class. Class features POMP have better alignment with centroids of the corre- sponding images, and are distributed with better uniformity.
generalization of the pre-trained prompt. Furthermore, we analyze the impact of the adaptive margin (5) in the local correction. We pre-train prompts on ImageNet-21 with varying values (0, 0.5, I, 1.5) and report the cross-dataset accuracy. Notably, our local correction method dynamically sets m to 1.5 when K = 319, and to when K 1000. The results are shown in Table 9 When the number of sampled classes relatively small (K 319), increasing the margin creates more space for potential negative classes, thereby improves cross-dataset ac- curacy. Conversely, for large K" values (e.g.. 1000), imposing very large margin (m = 1.5) disrupts the natural class distribu- tion and diminishes generalization ability. Overall, compared to the fixed margins [1] [54], our adaptive margin decreases as K increases, achieving optimal performance across different computing budgets (controlled by K) and sparing the time for extensive
Table 9: Ablation on the adaptive margin m in the local correction.
K-319
K-1000
0
65.2
65.8
0.5
65.7
66.2
67.0 (Ours)
1.5 66.5 (Ours)
66.4
search. See Appendix B for more ablation studies the number of shots
and prompt length.
4.5
Understanding the Pre-trained Prompt
To better understand the pre-trained prompt, we analyze the feature space of POMP through the properties of alignment and uniformity [56]. Intuitively, the image feature and its ground-truth class feature are supposed to stay closed (alignment). Besides, all the class features should be uniformly distributed to preserve maximal information and make the categories more distinguishable (uniformity). We use the alignment and uniformity loss the vision-and--ene and-lang field [45] 63) for representation probing. The alignment loss calculates the expected distance between features of an image x and its ground truth class w (@).
w (0))
align
E (x.))D
(6)
while the uniformity loss measures how well the class features w ()) are uniformly distributed:
Lumitorm A log
E EENEE
exp(-2/lw (9)
12)).
We visualize the alignment and uniformity measures of POMP and the previous SOTA, MaPLe, in Figure 5] For both measures, lower numbers are better. The circle of POMP in the figure is located in the lower left with the lightest color, indicating relatively smaller losses and the best performance under the cross-datased setting. Compared with the method without local correction, POMP significantly reduces the uniformity loss only slight expense of alignment. In other words, pre-trained prompt not only ensures the alignment of the image and the ground- trouhd-truth class, but

---

also disperses the class features in the representation space, thereby improving the generalization and robustness of the model. The visualization of the feature space Figure also verifies our superiority. The endpoints of the POMP class features are closer to the centroids of the image features, indicating better alignment and reduced aaignn loss (from 1.39 1.36 on Aircraft and from 1.41 to 1.36 on UCF101). Furthermore, the larger angles between the POMP class features demonstrate better feature uniformity and reduced (uniform loss (from -0.66 to -0.81 on Aircraft and from -0.95 to 1.23 UCFI01) compared to CoOp.
5 Conclusion
We present POMP pre-train general soft prompt on ImageNet-21K K for universal visual discrimina- tion. The learned prompt can be easily plugged into various visual recognition datasets and tasks for zero-shot inference. Experiments on open- vecablocyy image classification, semantic segmentation, and object detection show that POMP surpasses previous methods by a considerable margin.
Limitations
To facilitate future research, we analyze the limitations in our work and propose potential solutions. (1) We present the local contrast and use the loss within subsampled class set an empirical estimation for the expected contrastive loss within the full class set. However, the theoretical risk of such estimation is urged to be investigated. (2) ImageNet-2IK: comprises vast number of classes that are organized based on semantic structure. By leveraging the hyponym and hypernym relations provided by WordNet synsets, We believe that utilizing the semantic information holds the potential to further enhance performance. (3) Despite the excellent performance exhibited by pre-trained prompt, its interpretability poses significant challenge because the context vectors are optimized continuous space. We leave it as future work.
derive the parent class and list of child classes for each class.
References
[1] Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Neural Information Processing Systems, 2019.
[2] M Saiful Bari, Aston Zhang. Shuai Zheng, Xingjian Shi, Yi Zhu, Shafiq R. Joty. and Mu Li. Spt: Semi-parameen prompt tuning for multitask prompted learning. 2022.
[3] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014.
[4] Max Bucher, Tuan- Vu, Matthieu Cord, and Patrick Pérez. semantti segmentation. ArXiv, 055/1990.0000 17, 2019.
[5] Adrian Bulat and Georgios Tzimiropoulos. Language-ane aware soft prompting for vision & language foundation models. ArXiv, abs/2210.01 115, 2022.
[6] Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1209-1215 2016.
[7] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained rision-and languagg models. In European Conference Computer Vision, 2020.
[8] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In Neural Information Processing Systems, 2021.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
10

---

[10] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[11] Jiankang Deng, J. Guo, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. 20/9 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4685-4694, 2018.
[12] Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Costa, Cees G. M. Snoek, Georgios Tzimiropoulos, and Brais Martínez. Variational prompt tuning improves generalization of vision-language models. ArXiv, abs/2210.02390, 2022.
[13] Jian Ding, Nan Xue, Guisong Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentaiinn 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11573- 1 11582, 2021.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.
[15] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guo Chun Li. Learning to prompt for open-vocabulary object detection with vision-language model. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14064-14073, 2022.
[16] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-338. 2010.
[17] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. 2004 Conference on Computer Vision and Pattern Recognition Workshop, pages 178 178, 2004.
[18] Chengjian Feng, Yujie Zhong, Zequn Jic, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards vocabulcyyy detection using uncurated images. In European Conference on Computer Vision, 2022.
[19] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.1 13921, 2021.
[20] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. Proceedings of the 28th ACM International Conference on Multimedia, 2020.
[21] Agrim Gupta, Piotr Dollár, and Ross B. Girshick. Lvis: A dataset for large vocabulary instance segmentation. 2019 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5351-5359, 2019.
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9733 9065,9735222 2019.
[23] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[24] Patrick Helber, Benjamin Bischke, Andreas R. Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEF Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12:2217-2226, 2019.
[25] Olivier J. Hénaff, A. Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. ArXiv, abs/1905.091 099000 2019.
11

---

[26] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu. Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of- distribution generalization. 2021 IEEF/CVF International Conference on Computer Vision (ICCV), pages 8320-8322 8009, 2020.
[27] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt. and Dawn Xiaodong Song. Natural adversarial examples. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15257-15266, 2019.
[28] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
[29]
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. ArXiv, abs/2210.03117, 2022.
[30] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European Conference Computer Vision, 2019.
[31] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. 2013 IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[32] Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Kolunn and René Ranftl. Language- driven semantic segmentation. ArXiv, abs/2201. .03546, 2022.
[33] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. ArXiv,
2021.
[34] Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, 2014.
[35] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5196-5205, 2022.
[36] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. ArXiv, abs/ 306/1306.5151, 2013.
[37] George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38:39-41, 1995.
[38] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Loddon Yuille. The role of context for object detection and semantic segmentation in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 891 1918 2014.
[39] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding sanuugg-image pretrained models for general video recognition. In European Conference on Computer Vision, 2022.
[40] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008
[41] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pages 3498 3505. IEEE Computer Society, 2012.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 1021.
12

---

[43] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guidd guided dense prediction with context-aware prompting. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18061-18070, 2021.
[44] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019.
[45] Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, and Xu Sun. Delving into the openness of CLIP. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, jul 2023.
[46] Shuhuai Ren, Junyang Lin, Guangxiang Zhao, Rui Men, An Yang, Jingren Zhou, Xu Sun, and Hongxia Yang. Learning relation alignment for calibrated cross-modal retrieval. In Annual Meeting of the Association for Computational Linguistics, 2021.
[47] T. Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi for the masses. ArXiv, abs/2104 10970 2021.
Imagenet-21K pretraining
[48] Shuai Shao, Zeming Li, Tianyuan Zhang. Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8429 9438,,83,,118 2019.
[49] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision- Asioonlanguage models. ArXív, abs/2209.07511 2022.
[50] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. ArXiv, abs/1212.0402, 2012.
[51] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference Computer Vision, 2019.
[52] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1 0000000748, 2018.
[53] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/ 70b./776..,362,, 2017.
[54] H. Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jin Zhou, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5265-5274, 2018
[55] Haohan Wang, songwei Ge, Eric P. Xing, and Zachary Chase Lipton. Learning robust global representations by penalizing local predictive power. In Neural Information Processing Systems, 2019.
[56] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. ArXiv, abs/2005. 10242, 2020.
[57] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non paraametiiiic instance discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4733-3742, 2018
[58] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero- and few-label semantic segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8248 8257, 2019.
[59] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[60] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Delving into inter-image invariance for unsupervised visual representations. International Journal of Computer Vision, 130:2994- 3013, 2020.
13

---

[61] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision- lannual model. ArXiv, abs/21 12.14757, 2021.
[62] Lewei Yao, Runhu Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grame tine-grained interactive language-image pre-training. ArXiv, abs/2111.07788 2021.
[63] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. ArXiv, abs/2210.07225, 2022.
[64] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122-52 5130, 2017.
[65] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for Vision-language models. International Journal of Computer Vision, 130:2337 2348, 2021.
[66] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for dision-languiel models. 2022 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 16795-16804, 2022.
[67] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krahenbuhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference Computer Vision, 2022.
[68] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühh. Probabilistic two-stage detection. ArXiv, abs/2103.07461, 2021.
[69] Benjin Zhu, Junqiang Huang, Zeming Li, Xiangyu Zhang, and Jian Sun. Eqco: Equivalent rules for self-supervised contrastive learning. ArXiv, abs/2010.01929 2020.
14

---

A Pre-training Details
We conduct prompt pre- pre-training on the ImageNet-21K dataset (official winter 2021 released version We follow the processing methods in [47] which involves cleaning invalid classes, allocating 50 images per class for a validation split, and crop-resizing all the images to 224 resolution. We conduct all the experiments on 8 NxNiiiaa V100 GPUs. For pre-training, the learnable vector is randomly initialized by drawing from a zero-mean Gaussian distribution with a standard deviation equal to 0.02. We use the SGD optimizer with initial learning rate of 0.002, decayed by the cosine annealing rule. The batch size is 32, and the maximum epoch is 20.
For the mask proposal network and region proposal network pre-training, we strictly follow the settings of ZSSeg [6]] and Detic [67], respectively. Specifically, we take MaskFormer [8] with 101
23 the mask proposal network. We use AdamW optimizer with the initial learning rate of le-4, weight decay of le-4, a backbone multiplier of 0.1, and a poly learning rate policy with a power of 0.9. Besides, we take CenterNet2 [68] detector with ImageNet-21k pre-trained ResNet-50 47 as the region proposal network. We use an Adam optimizer with learning rate 2e-4. Other tricks like Federated Loss, repeat factor sampling, and large scale jittering are incorporated to further improve the performance. As with Detic, we leverage both region-level and image-level supervision. We always first train a converged base-class-only model (4x schedule) and fine-tune with additional image-labeled data for another 4x schedule.
B Setting for Segmentation and Detection
Table outlines the settings for semantic segmentation and object detection. We further introduce the settings in detail from three perspectives: backbone, data processing, and prompt.
Backbone. In general, we adopt a two-stage framework for these two tasks. At stage one, we use pre-trained proposal network to generate a set of mask or region proposals. At stage two, classify each proposal with the class features generated by our POMP prompt. For semantic segmentation, our POMP shares the same visual backbone as ZSSeg 61], which uses pre-trained MaskFormer [8] with ResNet-101 (23) as default backbone to extract a set of binary masks. For object detection, our
POMP shares the same visual backbone with Detic (77, which takes CenterNet2 [68] detector with
ResNet-50 as its backbone, and leverages both region-level and image-level supervision.
67 to designate data belonging to two class sets as source data and target data, respectively. The proposal networks are pre-trained on the source data with the source class set, while conducting zero-shot evaluation on the target data with the target class set. There are two protocols for the source-target data split. The first is the open-vocabulary protocol, where the class set of one dataset is divided into two disjoint groups for the source and target data, respectively. The second protocol is the cross-dataset protocol, in which the source and target data are from two independent datasets with potentially overlapping class sets.
Data Processing. We follow previous work 58
61
19
We introduce the details of class set splitting in the ooca-locay protocol. COCO Stuff and Pascal VOC 2012 are the two semantic segmentation datasets using the voccbbbbyy protocol. Following previous settings (58 61, total of 171 annotated classes in COCO Stuff are divided into 156 seen classes and 15 unseen classes. For Pascal VOC 2012, a total of 20 classes are divided into 15 seen classes and 5 unseen classes, and the provided augmented annotations
used. LVIS is the object detection dataset using the open-vocabulary protocol. The standard LVIS dataset contains object detection and instance segmentation labels for 1203 classes. The classes are divided into three groups: frequent, common, and rare, based on the number of training images. According to previous work 19) the data from the 866 frequent and common classes are considered the source data, while those from the remaining 337 rare classes are the target data in testing. We note that Detic utilizes both superviedd data from LVIS as well as mmage-supervisee data from ImageNet-22 1 that overlaps with LVIS (997 classes, 277 of which are novel classes). This allows Detic to demonstrate transfer not only from base to novel classes, but also from image-level to box-level recognition. Since Detic the closest existing method to ours that leverages ImageNet-21 K, we chose it as strong baseline and followed its setup for fair comparison.
15

---

Table 10: Settings for semantic segmentation and object detection.
Souree Data and Class Set (for proposal network pre-triining)
Target Data and Class Set (for zcro-shor evaluation)
Tisk
Proposal Network
Sctting
Semantic
MaskFormer
Open-vocah COCO Stuff
COCO Stuff (seen)
COCO Stuff (unseen)
Seguentation
(Mask Proposal Network)
Open PASCAL VOC
PASCAL VOC (seen)
PASCAL VOC (unseen)
Cross-ataser
COCO Staff
ADE20K PASCAL Context
Object
CenterNet2 detector
Open-vocab LV18
LVIS (hogaent+common)
ImageNet- (overlaps with LVIS)
LVIS (rare)
Detection
(Region Proposal Network)
Cross-dataset
LVIS
COCO/ / Object365
ImageNet-21KK (overlaps with LVIS)
Prompt. 7SSeg provides two kinds of prompts: hand-crafted prompts and learning-based prompts. Hand-crafted prompts include single prompt, i.e., "a sculpture of a [CLASSNAME]", as well as ImageNet prompts [42 and ViLD prompts [19], which are used for prompt ensemble and consist of 80 and 14 hard prompts, respectively. The learning- baasd prompt is obtained by fine-tuning a randomly initialized soft prompt on the source data. Accordingly, for fair comparison, of experiments based on whether to use the source data for prompt fine-tuning (1) The results of ZSSeg with various hard-crafted prompts and the pre-trat me--trinee POMP prompt without access to the source data can be found in Table 13 in Appendix E.3 (2) The results of ZSSeg with learning- basse prompts initialized from random vectors and our pre-trained POMP prompt, both using source data for further fine-tuning, can be found in Table and Table 5 in & 4.3.2. Detic has also extensively delved into intricate prompts, such "a photo of [CLASS] in the scene". Moreover, it has made endeavors to employ synonyms for each category. Nevertheless, its ultimate recommendation to use simple yet effective prompt, i.e., "a [CLASSNAME]", and all its released checkpoints are based on this prompt. We strictly adhere to Detic's best practice, the evaluation of Detic and POMP in 4.3.3 are both conducted without any further prompt tuning on the source data.
conducted two sets
C Datasets
The details of the downstream datasets for image classification, semantic segmentation, and object detection are shown in Table
Image Classification. For cross-dataset image classification, we evaluate the performance of POMP
on 10 downstream datasets, including Caltech-101 [7]. Oxford-Pets [41], Stanford Cars (31), Oxford-
Flowers [40]. Food-101 (3), FGVC Aircraft [36). EuroSAT (24), SUN-397 [59] Describable
Textures (DTD) [9], UCF-101 [50]. We also conduct zero-shot evaluation on 4 out-of datasets
derived from ImageNet [10), including ImageNetV2 [i, ImageNet-S (55), ImageNet-A 27). and
ImageNet-R (26). to evaluate the domain generalization capability of our method.
Semantic Segmentation. We perform open-vocab semantic segmentation on COCO Stuff 6 and Pascal VOC 2012 [66. Following previous notation and settings [58] 61, we split the class set into seen and unseen classes, where data for seen classes is considered the source data and data for unseen classes is considered the target data. The major measures for evaluation include mloU and the harmonic mean IoU (hloU) among both seen and unseen classes [61]. The hloU is defined as:
hIoU 2 x mloUseon mloU unscen
mIoUsee + mIoU Inssen
We also conduct cross-dataset evaluation, which takes the standard COCO Stuff dataset as the source dataset for pre-training mask proposal network, and then conducts zero-shot inference on ADE20K [64] and PASCAL Context (38)
Object Detection. We evaluate the performance of POMP on the object detection dataset LVIS 21 under the open-vocabulary setting proposed by [99. The source data consists of bexelevel data from LVIS's 866 frequent and common classes, as well as image-level data from ImageNet-22 2 that overlaps with LVIS. The target data for testing comprises the remaining 337 rare classes in LVIS. We take AP,, i.e., AP on rare classes, as the major measure. AP, and AP,. i.e., AP on frequent and common classes, are also reported. In the cross-dataset setting, the region proposal network is
16

---

Table 11: Datasets in experiments.
Dataset
Classes
Train Size
Test Size
Metric
Datasets of Image Classification
Caltech-101 101
[1
102
3,060
6,086
mean per-class accuracy
Oxford-IITT Pets 41
37
3,680
3,669
mean per-class accuracy
Stanford Cars 31
196
8,144
8,041
accuracy
Oxford Flowers- 102 40
102
2,040
6,149
per-class accuracy
Food-101 101
8
101
75,750
25,250
accuracy
FGVC Aircraft 136
100
6,667
3,333
mean per-class accuracy
SUN-397 59
397
15,880
19,850
accuracy
Describable Textures (DTD) 9
47
3,760
1,880
accuracy
EuroSAT 24
10
10,000
5,000
accuracy
UCF-101 550
101
7,639
3,783
accuracy
ImageNetV2 44
1,000
10,000
10,000
accuracy
ImageNet-S
55
1,000
50,889
50,889
accuracy
ImageNet-A
27
200
7,500
7,500
accuracy
ImageNet-R
[26
200
30,000
30,000
accuracy
Datasets of Semantic Segmentation
COCO Stuff [6]
171
117K
5K
mloU (scen/unseen), hloU
PASCAL VOC [16
20
11,185
1,449
mloU (seen/unseen), hloU
ADE20K 64
150
20K
3K
mloU, fwloU, pACC mloU, fwloU, PACC
PASCAL Context 38
59
10,103
9,637
Datasets of Object Detection
LVTS 21
1,203
100,170
19,822
AP,, APc. AP,, AP
COCO 34
80
118K
5K
AP, APs0. AP75. AP,. AP AP,
Object365 48
365
600K
38K
AP, APs0, AP7s, APs, APms AP,
pre-trained on the source dataset, which includes standard LVIS and ImageNet-21K (overlapping with LVIS). It is then directly used for inference target datasets: COCO (34) and Object365 [48 We use AP, AP50, AP75, APs, APm. and AP; the evaluation metrics.
D Qualitative Results for Semantic Segmentation and Object Detection
In this section, we provide more qualitative results our POMP for semantic segmentation and object detection. F shows another three cases on open- vocabulary COCO-Stuff segmentation. POMP demonstrates a stronger ability than ZSSeg in the recognition of background classes. In case (1), POMP correctly identified the dirt and plant-other in the scene, instead of marking all these areas as grass. In case (2) and (3), POMP recognizes the classes of clouds and tree, respectively, while ZSSeg misclassifies them sky-other and bush. However, POMP misses some objects of sheep located at the edge in case (2) and neglects the object of branch in case (3), indicating it still has insufficient recognition of small objects. For object detection, Figure Dillustrates qualitative results on LVIS images. Base and novel categories are shown in purple and green, respectively. POMP identifies regions from the novel class without using the corresponding 1 I.2K detection annotations, demonstrating its generalization in the wild.
E
More Ablation Study
E.1
Ablation on Proposal Distribution
As introduced $ 3.2, we also investigate other types of proposal distribution for local contrast and negative class sampling. The first is the frequency distribution Q(J), which samples the negative class based on the number of training samples belonging to this class. Note that the original ImageNet-21K is class-imbalanced, i.e., the number of training samples belonging to common classes is larger than those belonging to rare classes, which can roughly reflect the long-tail distribution of object categories in nature. The frequency distribution will allow for more sampling of common classes while suppressing the exposure of rare classes in prompt tuning, Let M, be the number of
17

---

prass
g7355
opsta
BTRSS
bush
pirafie
ZSSeg (Baseline)
POMP (Ours)
Ground-truth
Figure 7: More qualitative results on open-vocabulary COCO-Stuff segmentation.
Figure 8: Qualitative results on LVIS images. Base and novel categories are shown in purple and
green colors respectively. We use a score threshold 0.5 and show the most confident class for each box.
18

---

Table 12: Ablation on the sampling distribution in POMP based on CLIP (ViT/B-16) backbone.
Cross-dataset ImageNet-21K
Cross-domain (4 Avg.)
Method
(10 Avg.)
POMP (uniform distribution)
25.3
67.0
60.8
POMP (frequency distribution)
(-.94-0.4)
66.2
60.1
РОMP (similarity distribution)
(-.67
64.2
(-.26)
training samples belonging to the negative class i, the frequency distribution is defined
M2
(8)
25=1 M,
The second is the similarity distribution Q(+), which aims to sample more hard negative classes. Hard negative classes are those that have a higher similarity between their features and the features of the input images, and more likely to be confused with the positive class. Accordingly, in the similarity distribution, the likelihood of a negative class being sampled increases as the similarity between its feature and the image feature increases. To achieve this, we pre-encode features of all classes represented by a hand-crafted prompt (i.e.. "a photo of [CLASSNAME]"), The feature of class i is denoted as w,. The likelihood of sampling a negative class is determined by the similarity between the class feature w; and the image feature x:
exp(x T wr/T)
e.0¹ (x)
Σ exp(x w//T)
Table 12 illustrates the performance of different proposal distributions. Compared to the uniform distribution, using the frequency distribution for sampling leads to degraded performance, particularly in cross-dataset and cross-domain settings, due to reduced sampling of rare categories. This highlights the importance of large number of long-tail categories in the ImageNet-21 dataset for the generalization of the soft prompt. Additionally, the performance of the similarity distribution is also not as strong as that of the uniform distribution. The reason for this may be that as the soft prompt evolves, the features of hard negative classes change. However, the negative features used in (9) are obtained from the hard prompt, creating fixed proposal distribution that is unable adapt to these changes, potentially causing the soft prompt to converge to local optimum. In contrast, POMP with the simple uniform distribution considers both common and rare classes, as well easy and difficult classes, leading to the best performance for both the soft prompt and class features.
E.2 Ablation on #shot and Prompt Length
We further conduct ablation the number of pre-training instances per class (#shot) and the prompt length to analyze their influence on the generalization ability of POMP. The left panel in Figure 9 illustrates the results of #shot. The green curve represents the average accuracy of 10 datasets under the cross-dataset evaluation, while the purple curve represents the avergaed accuracy of 4 datasets under the cross-domain evaluation. Overall, the performance of POMP improves #shot increases. We find that POMP can achieve decent cross-dataset and cross-domain accuracy even with #shot=1. 1. This is due the huge number of classes in ImageNet-22 I.. Even if there are only one instance per class, the overall amount of data (21K instances for 21K classes) is enough for training a soft prompt with only 0.012 M learnable parameters.
The right panel in the figure shows the results of the prompt length. The soft prompt of length 16 achieves 65.0% accuracy across datasets, which is lower than the soft prompt of length 4 with 67.0% cross-dataset accuracy. It indicates that the prompt with too large lengths impairs its generalization, which consistent with the findings from previous work [66]] 29
E.3 Ablation on Prompt Types for Semantic Segmentation
We perform an ablation study on prompt types for cross-dataset semantic segmentation to further demonstrate the superior generalization ability of our prompt on downstream tasks. Specifically, we
19

---

Avg.)
pross-datased (10 Avg.)
Accuracy (%)
66
66
66
65
64
64
2
16
Accuracy (%)
60.5
60.5
60.0
60.0
59.5
59.5
Cross-domain Avg.)
drossndemain Avg.)
59.0
59.0
2
16
8
16
#shot
prompt length
Figure 9: Ablation study on #shot and prompt length. When varying #shot, the prompt length and when varying the prompt length, #shot is 16.
Table 13:
Cross-dataset evaluation for semantic segmentation. All methods share the same visual backbone with ZSSeg, but use different prompts.
Source Datasct:
Target Ditaset: ADE20K
Target
Method
Stundard COCO Stuff
PASCAL Comext
mioU
twiloU
mACC
pACC
fwloU
mACC
pACC
mioeU
fwloU
mACC
pACC
2SSog (single prompt)
40.5
47.8
53.5
44.0
52.9
51.8
64.6
74.3
2SSeg (ImageNet prompts)
40.9
48.4
54.7
17.7
46.5
31.8
57.1
52.0
64.7
70.3
75.4
ZSSeg (ViLDD prompts)
40.9
48.6
54.2
62.3
20.2
49.1
33.4
60.7
51.8
63.8
69.6
73.8
ZSSog
prompt, ours)
take 7SSeg as the backbone and evaluate the performance of four types of prompts, as descrihed in Appendix B As shown in Table [3] ZSSeg with our POMP prompt achieves the highest performance on the three datasets. It is noteworthy that, despite using 80 hard prompts for ImageNet prompts and 14 for ViLD prompts for prompt ensemble, their performance was consistently worse than our POMP with just one soft prompt, highlighting the effectiveness of our method.
20


Reviews:
[{"review_id": "tZhDMP75zh", "reviewer": "Reviewer_9929", "summary": "This paper introduce POMP, a prompt pre-training method for vision-language models. POMP learns a universal soft prompt that can express a large number of visual concepts and transfer to various visual recognition tasks in a zero-shot manner. POMP uses local contrast and local correction strategies to reduce the training cost and improve the generalization of the prompt. The paper shows that POMP outperforms previous state-of-the-art methods on several datasets and tasks, such as image classification, semantic segmentation, and object detection. The paper also analyzes the feature space of POMP and demonstrates its alignment and uniformity properties.\n", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It proposes a novel and efficient method to pre-train a soft prompt on a large-scale dataset with over twenty-thousand classes, which enables the prompt to capture rich semantic information for visual recognition.\n2. It introduces local contrast and local correction techniques to reduce the computational and memory overhead of prompt tuning\n3. It achieves state-of-the-art performance on 21 datasets for various vision tasks, such as image classification, semantic segmentation, and object detection.", "weaknesses": "1. The training process for Stage 1 is similar to that of CLIP; however, CLIP utilizes a significantly larger pre-training dataset compared to ImageNet 22k. Therefore, it is important to clarify why the CLIP model cannot directly excel at these downstream tasks and why tuning it on ImageNet 22k could yield better results. Currently, it is unclear whether the performance improvement is primarily attributed to ImageNet 22k or the proposed prompt pre-training methods.\n2. Figure 2 may inadvertently create misconceptions by suggesting that training solely on ImageNet 22k enables zero-shot transfer to tasks such as object detection and segmentation. In reality, as explained in the supplementary material, after completing Stage 1 training, an additional round of pre-training on the source data specific to object detection and segmentation is necessary. To avoid confusion, it is essential to provide explicit details regarding this additional training process.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We hope that our response can address your concerns and you can consider our work more favorably.\n\n>**Q1: POMP’s pre-training process for stage 1 is similar to that of CLIP, why can't CLIP directly excel at downstream tasks compared to POMP, and why does tuning on ImageNet-22K yield better results?** \n\nA1: While POMP and CLIP both pre-train on image-text pairs, **there is a key difference in the text formatting.** CLIP uses paired captions, while POMP uses a \"soft-prompt + [CLASSNAME]\" format. This leads to two issues for CLIP when transferring to downstream tasks:\n1. **CLIP requires extensive prompt engineering**, crafting hard prompts like \"a photo of a [CLASSNAME]\" for synthesizing classification weights. But hard prompts are unstable, as addressed in previous studies [1], small wording tweaks can drastically impact performance. For example, for the Caltech101 dataset, changing the prompt from “a photo of a [CLASS]” to “a photo of [CLASSNAME]” causes more than 5% decrease in accuracy.\n2. **Generic prompts of CLIP lack task-relevant context**. For fine-grained/long-tailed datasets, prompts without contextual clues achieve lower accuracy. For example, for the EuroSAT dataset, the prompt of “a photo of a [CLASSNAME]” achieves 13% less accuracy compared to “a satellite photo of [CLASSNAME]”. \n\nIn contrast, by pre-training a universal soft prompt on ImageNet-22K, POMP adapts better to downstream tasks without prompt engineering. The pre-trained prompt encodes broad coverage of visual concepts, providing more expressive context (see Line 252-254). This allows POMP to outperform CLIP on downstream transfer without the need for extensive fine-tuning.\n\n[1] Learning to Prompt for Vision-Language Models. Zhou et al. IJCV 2022.\n\n>**Q2: Avoid potential confusion in Fig.2**\n\nA2: Thanks for your suggestion. As you note, for detection and segmentation tasks, the region proposal and mask proposal networks need to be pre-trained on detection and segmentation data, respectively, and the hard prompt (e.g., “a photo of a”) should be replaced with our POMP prompt. We have updated the caption of Fig.2 with these notes in our final revision (see our attached pdf). "}, {"review_id": "gySlAXVmfZ", "reviewer": "Reviewer_i3py", "summary": "This paper proposes POMP, a prompt pre-training method for pre-trained vision-language models like CLIP. While existing prompt-tuning approaches usually fine-tune the soft prompts on a specific downstream dataset with a limited number of classes, the proposed POMP conducts prompt \"pre-training\" on a large dataset (i.e., ImageNet-21K) with much more classes. POMP consists of two strategies: local contrast and local correction -- the former makes the memory footprint affordable with common GPUs, while the latter reduces the bias caused by the class sampling. Experiments show that POMP outperforms previous prompt-tuning methods across various visual recognition tasks and datasets.", "questions": "This paper makes solid contributions to the prompt tuning of pre-trained vision-language models. The proposed method is simple and well-motivated, and shows superior performance on a wide range of visual recognition tasks and datasets. The presentation of the paper is clear, with easy-to-understand figures and equations. Some minor suggestions are listed in the weaknesses section. ", "limitations": "The limitations are discussed in the appendix of the paper. However, there is no discussions about the potential social impact, although the corresponding entry is checked in the checklist. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Extending prompt-tuning of vision-language models from task-specific datasets to a large \"pre-training\" dataset (e.g., the ImageNet-21K in this paper) is a promising direction, as existing prompt-tuning methods demonstrate poor cross-dataset generalization ability and always require per-task training. \n\n- The proposed method is simple and has natural intuitions. The local contrast strategy makes the prompt tuning feasible with a large number of classes, but the aggressive class sampling introduces potential bias. Thus, the local correction is further proposed to alleviate the negative impact of the sampling bias. The definition of the adaptive margin m is clever and provides good insights. \n\n- The experiments of this paper have wide coverage, which includes cross-dataset and cross-domain image classification, open-vocabulary object detection, and open-vocabulary semantic segmentation. The results can provide good references for future works in prompt tuning. Furthermore, the proposed POMP shows superior performance across various tasks and datasets, establishing a new state of the art for prompt tuning. \n\n- The code of the proposed method and all baselines are included in the supplementary material.\n", "weaknesses": "I don't find significant flaws in this paper. There are some minor suggestions:\n\n(1). Fig.1 looks a bit visually exaggerated: POMP only marginally improves the previous SOTA on some tasks and datasets, while the non-uniform scale in Fig.1 has a potentially misleading effect. \n\n(2). Investigating the embedding space of POMP from the perspective of alignment and uniformity is interesting, but it is hard for me to parse \"Class features of POMP have better alignment with centroids of the corresponding images, and are distributed with better uniformity\" from Fig.6. Maybe it would be better to also add the concrete values of the alignment and uniformity loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive assessment and constructive feedback. \n\n>**Q1: The non-uniform scale in Fig.1 has a potentially misleading effect.**\n\nA1: Thanks. Considering the tasks covered in the radar chart (Fig.1) have different difficulties and use different metrics (e.g., acc for classification, hIoU for segmentation), we follow previous work [1,2] to use independent scale for each task. To reduce potential misleading, we have updated the figure to use a consistent scale of 0.46 for all tasks except “Semantic Segmentation (Open-vocab PASCAL VOC)”. Given the large performance gain (+6.9) and range for this task, we have removed it from the figure to avoid compressing the other tasks. Please refer to our attached pdf.\n\n[1] CoCa: Contrastive Captioners are Image-Text Foundation Models. Yu et al. TMLR 2022.\n\n[2] Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks. Wang et al. CVPR 2023.\n\n> **Q2: Avoid potential confusion in Fig. 6.**\n\nA2: Thanks for your suggestions. In Fig.6, our intention is to convey that: \n1. Compared to light dashed lines (CoOp), the end points of solid lines (POMP) is closer to centroids of the points (image features), indicating that “class features of POMP have better alignment with centroids of the corresponding images''. \n2. Compared to light dash-dot lines (CoOp), the angle between solid lines (POMP) is larger, indicating that “class features of POMP are distributed with better uniformity”. \n\nAs you said, adding concrete values of the alignment and uniformity loss is a good solution. For Aircraft, POMP reduces the alignment loss $\\ell_{\\text{align}}$ from 1.39 to 1.36 and the uniform loss $\\ell_{\\text{uniform}}$ from -0.66 to -0.81. For UCF101, POMP reduces $\\ell_{\\text{align}}$ from 1.41 to 1.36 and $\\ell_{\\text{uniform}}$ from -0.95 to -1.23. We have updated the figure in our final version, please refer to our attached pdf.\n"}, {"review_id": "5GYH8kVeDp", "reviewer": "Reviewer_Mx1X", "summary": "* This paper present a prompt pre-training method for vision-language models named as POMP, which can be transfer to visual recognition tasks including image classification, semantic segmentation, and object detection.\n\n* POMP follow the prompt tuning setting of CoOp. The authors claims that pre-training prompts on ImageNet-21K requires over 300 GB GPU memory with traditional methods like CoOp. To make it work and more efficient on ImageNet-21K, POMP propose a positive and negative class sampling to reduce gpu memory cost.\n\n* POMP is prompted learning on ImageNet-21K dataset, also is also a task-agnostic prompt pre-training method, which can be directly used in downstream methods.", "questions": "please refer to weakness", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "* The paper is well organized, the framework is simple.\n\n* The authors have conducted a lot of experiments on downstream tasks.\n\n* The design of local contrast and local correction is memory efficient for prompt tuning.", "weaknesses": "* The peformance of open vocabulary object detection & instance is really not high compared with detic(24.9 -> 25.2), even though POMP is prompt tuning on ImageNet21K dataset. The observation can be also validated in Table 6(Cross-dataset evaluation for object detection), For me, it seems that POMP don't work on detection benchmark.\n\n* On detection experiments, POMP is based on detic, however, detic and POMP is not strictly open vocabulary method,  which means that the novel class names is known in training phase. For standard  open vocabulary tasks, the novel class can not be seem or trained, so the novel class can not be prompt tuned in this setting, How does POMP cope with this problem ? \n\n* For me the novelty and the technical contribution of this paper is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We humbly think that some concerns are caused by misunderstandings, which we will explain in detail below. We hope that our response can clarify the misunderstandings so that you can consider our work more favorably.\n\n>**Q1: The performance gain on the object detection tasks is limited.**\n\nA1: We note that the +0.3 mAPr gain on open-vocabulary LVIS is meaningful since Detic is a very strong baseline which uses extra ImageNet-21K data for detector pre-training. More importantly, on cross-dataset detection (Table 6), our POMP shows sizable gains of +1.9 AP50 on COCO and +0.8 AP50 on Object365. \n\nWe would like to reiterate our main contribution that our method enables large-scale prompt learning for the first time, **rather than optimize detection scores**. We pre-train a universal prompt on ImageNet-21K that transfers broadly, achieving SOTA on 21 downstream tasks in zero-shot settings: the consistent gains across classification, detection, and segmentation tasks have offered strong evidence of its effectiveness.\n\n>**Q2: POMP takes Detic as backbone model for detection experiments. However, Detic is not a strictly open-vocabulary method for LVIS, how does POMP cope with this problem?**\n\nA2: Thanks for your feedback. We want to clarify our rationale for choosing the Detic backbone and experimental setup, and demonstrate that our improvement is consistent across various settings.  \n1. For the open-vocabulary LVIS task, Detic utilizes both box-supervised data from LVIS-base (866 classes) as well as image-supervised data from ImageNet-21K that overlaps with LVIS (997 classes, 277 of which are novel classes). This allows Detic to demonstrate transfer not only from base to novel classes, but also from image-level to box-level recognition. Since Detic is the closest existing method to ours that leverages ImageNet-21K, we chose it as a strong baseline and followed its setup for fair comparison.\n2. To evaluate Detic and POMP more strictly on open-vocabulary LVIS, we conduct additional experiments training only on LVIS-base 866 classes using box-supervised data alone (denoted as **LVIS-base**. In contrast, Detic’s original setting is denoted as **LVIS-base & IN-L**). The results are as follow:\n| Method | Source training data | mAPr |\n|:-------:|:---------------:|:-----:|\n| Detic | LVIS-base | 16.4 |\n| POMP | LVIS-base | **17.4** |\n| Detic | LVIS-base & IN-L | 24.9 |\n| POMP | LVIS-base & IN-L | **25.2** |\n\nAs shown, our method still shows consistent improvements over Detic, demonstrating the general efficacy of our approach for open-vocabulary detection. We would be happy to include these additional experiments in the final paper. Please let us know if you would like any clarification or have additional suggestions.\n\n3. Additionally, we would like to highlight that our **cross-dataset experiments** (Table 6), training on LVIS-full and evaluating on COCO and Object365, **follow a strictly open-vocabulary protocol**.  As we reponsed in A1, our POMP achieves SOTA performance in these experiments as well, with gains of +1.9 AP50 on COCO and +0.8 AP50 on Object365 compared to prior art. In conclusion, we have provided strong evidence through additional experiments and analyses that our method advances the state-of-the-art in open-vocabulary detection across multiple datasets and protocols.\n\n>**Q3: The novelty and technical contribution are limited.**\n\nA3: We appreciate your feedback and want to clarify that our work makes several key innovations. First, our proposed methods of local contrast and local correction are novel techniques for efficient prompt learning at scale. More importantly, it is the combination and application of these techniques that enables prompt learning on large-scale visual concepts (e.g. ImageNet-21k) for the first time. This is a major contribution given that previous prompt tuning methods cannot feasibly scale to such a large number of concepts due to their computational complexity. \n\nBy pre-training a universal prompt on ImageNet-21K, we obtain a prompt representation that transfers broadly and achieves SOTA results on 21 downstream tasks spanning classification, detection, and segmentation. We hope that our responses can clarify the misunderstandings and you will consider increasing your scores after seeing our responses. We'd appreciate any references or literature you can suggest, as we aim to cite them in our work.\n"}, {"review_id": "L2RRFKT27s", "reviewer": "Reviewer_xVVp", "summary": "This paper proposed a vision-language pre-training method called POMP. It aims to solve the GPU memory problem of the existing vision-language pre-training method like CoOp when the class number is extremely huge. POMP is composed of local contrast and local correction strategies. Local contrast is to sample a few classes during training to save the GPU memory, while local correction is to alleviate the sampling bias problem. The authors provide very comprehensive experiments to show the effectiveness of the proposed pre-training method.", "questions": "None.", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed local contrast strategy is reasonable, easy to implement, and effective when the class number is huge.\n2. The experiments are very comprehensive, and the code is also provided. So this work could provide valuable contributions to the open-source community.", "weaknesses": "The proposed local correction strategy is more like a general approach in the contrastive learning method. It encourages the positive samples to be much closer and pushes negative samples away for a more stringent decision boundary. So the reason why it works is that the features are more uniformly distributed in the feature space, which is supported by Fig. 5. Therefore, I do not think that the authors' claim that this strategy solves the sampling bias problem is proper, since the sampling bias still exists. Therefore, I encourage the authors to rethink the motivation and reason of the contrast correction strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive assessment and insightful comments. \n\n>**Q1: The claim that the local correction strategy solves the sampling bias problem is questionable.**\n\nA1: Thanks for your feedback. We agree that sampling bias still exists with our approach. However, our intention is to convey that the local correction helps mitigate negative effects stemming from sampling bias. As we describe in Section 3.2.2, class sampling causes the absence of some negative classes, which further leads to a more loose decision boundary and less-uniform class feature distribution. \n\nTo alleviate this problem, our local correction strategy adds a margin term to create space for unsampled classes in the feature space. This encourages more discriminative representations even without observing all negative classes, which reduces the uniform loss from -1.05 to -1.37 and improves accuracy from 65.8 to 67.0 on ImageNet-21k (Fig.5). We will refine our statement of the local correction strategy in the final version. \n"}]

Author rebuttal:
We want to thank all the reviewers for the positive assessment and insightful comments, which helped improve the quality of our work. We have revised our paper accordingly and provided individual responses to each reviewer. Please find attached a PDF outlining the main changes:

1. Updated Fig.1 with a consistent scale.  
2. Clarified in Fig. 2 caption that the region/mask proposal networks require pre-training on detection/segmentation source data, as suggested. 
3. Added concrete values of the alignment and uniformity loss in Fig. 6. 

We hope that our responses can clarify the misunderstandings and help the reviewers consider our work more favorably.


Meta-review: Reviewers have come to a consensus on accepting this paper, the authors are expected to incorporate the suggestions from reviewers in the final camera ready version.
Decision: Accept (poster)