SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cross-dataset and Cross-domain Image Classification. Our POMP prompt, which has been pre-trained on a large number of classes, demonstrates a strong generalization ability. As shown in Table 2, POMP achieves the highest average accuracy of  $67.0\%$  when transferred to 10 downstream image classification datasets, outperforming CoOp by  $3.1\%$  and surpassing the previous SOTA in
Table 2: Cross-dataset and cross-domain evaluation for image classification. The backbone is ViT/B-16. Overall, POMP achieves the highest average accuracy, indicating better generalization.
Target (cross-dataset)	Target (cross-domain)
Caltech101	O2oortPeis	StanfordCars	Flowers102	Foot101	Aircraft	SUN397	DTD	EuroSAT	UCF101	Average	ImageNetV2	ImageNetS	ImageNet-A	ImageNet-R	Average
hard prompt	93.3	88.2	65.6	67.4	85.3	23.7	62.6	44.3	42.0	65.1	63.7	60.9	46.1	47.8	74.0	57.2
CoOp [65]	93.7	89.1	64.5	68.7	85.3	18.5	64.2	41.9	46.4	66.6	63.9	64.2	48.0	49.7	75.2	59.3
CoCoOp [66]	94.4	90.1	65.3	71.9	86.1	22.9	67.4	45.7	45.4	68.2	65.7	64.1	48.8	50.6	76.2	59.9
LASP [5]	94.5	89.4	64.8	70.5	86.3	23.0	67.0	45.5	48.3	68.2	65.8	63.8	49.0	50.7	77.1	60.1
VPT [12]	93.7	90.6	65.0	70.9	86.3	24.9	67.5	46.1	45.9	68.7	66.0	64.2	49.2	51.3	77.0	60.4
MaPLe [29]	93.5	90.5	65.6	72.2	86.2	24.7	67.0	46.5	48.1	68.7	66.3	64.1	49.2	50.9	77.0	60.3
POMP (Ours)	95.0	89.5	66.8	72.4	86.3	25.6	67.7	46.2	52.1	68.5	67.0	63.8	49.8	51.6	77.9	60.8
7/10 datasets. This is due to the fact that, after learning with enormous long-tail categories, POMP can provide a more expressive context for fine-grained visual concepts such as specific objects and scenes, resulting in improved performance on datasets like StanfordCars  $(+1.2\%)$  and Aircraft  $(+0.9\%)$ , as well as SUN397  $(+0.7\%)$  and EuroSAT  $(+4\%)$ . Furthermore, POMP is more robust to domain shift and achieves a new SOTA with  $60.8\%$  accuracy on 4 out-of-domain variants of the ImageNet dataset.
Training Efficiency. POMP also achieves comparable accuracy to the classical prompt tuning methods when fine-tuning on specific downstream datasets, but significantly reduces the training cost. Table 3 shows the performance of prompt tuning on ImageNet-1K, using a visual backbone of ViT-B/16 and 16 shots. The epoch is 50 for CoOp and POMP, and 10 for CoCoOp. CoOp generates all the 1000 class features at each training step, which consumes 28 GB of memory and takes 5.9 hours to finish the fine-tuning. The training time of CoCoOp is
Table 3: Prompt tuning on ImageNet-1K. POMP  $(K = 128)$  achieves comparable accuracy with CoOp and CoCoOp, but using less than  $19\%$  GPU memory and  $50\%$  training time.
Method	Acc. (%)	GPU Mem. (GB)	Training Time (h)
CoOp	71.9	28.2	5.9
CoCoOp	70.1	28.3	27.5
POMP (K=128)	71.2	5.3	2.7
POMP (K=256)	71.4	8.8	3.3
POMP (K=512)	71.6	15.9	4.2
even longer because it devises instance-specific prompts that require an independent forward pass for each image. Compared to these baselines, POMP ( $K = 128$ ) achieves competitive accuracy on ImageNet-1K while using less than  $19\%$  of GPU memory and  $50\%$  of training time, demonstrating its superiority.
4.3.2 Open-Vocabulary Semantic Segmentation
Table 4 shows the results of our method on open-vocabulary COCO Stuff and Pascal VOC. POMP outperforms the previous state-of-the-art method, ZSSeg [61], with a higher hIoU of 39.1 and mIoU-unseen of 38.2 on COCO Stuff. On Pascal VOC, the improvement of POMP is more significant with  $+6.9$  hIoU and  $+4.3$  mIoU-unseen. Figure 4 illustrates qualitative results on open-vocabulary COCO-Stuff, where POMP demonstrates a stronger ability to distinguish background categories compared to ZSSeg. For example, in case (1), ZSSeg misclassifies the classes of playingfield as dirt, while the POMP prompt with richer contextual semantics better expresses the difference between regular land and the playingfield with specific textures, thus facilitating the matching of the visual region with the ground-truth class.
POMP also demonstrates its generalization ability in cross-dataset settings. Taking standard COCO Stuff as the source dataset for mask proposal network pre-training, POMP achieves 20.7 mIoU and 51.1 mIoU when transferred to the target datasets of ADE20K and PASCAL Context, respectively, outperforming ZSSeg by +1.3 mIoU and +0.3 mIoU. Overall, POMP obtains remarkable gains over previous works in all settings.
4.3.3 Open-Vocabulary Object Detection
Table 4: Comparison with state-of-the-art methods on COCO Stuff dataset and Pascal VOC dataset. POMP and ZSSeg share the same mask proposal network and training strategy.
Method	Open-Vocab COCO Stuff	Open-Vocab Pascal VOC
hIoU	mIoU	hIoU	mIoU
seen	unseen	seen	unseen
SPNet [58]	16.8	20.5	14.3	21.8	73.3	15.0
ZS3 [4]	15.0	34.7	9.5	28.7	77.3	17.7
CaGNet [20]	18.2	35.5	12.2	39.7	78.4	25.6
ZegFormer [13]	34.8	36.6	33.2	73.3	86.4	63.6
ZSSeg [61]	37.8	39.3	36.3	77.5	83.5	72.5
POMP (Ours)	39.1	39.9	38.2	84.4	93.6	76.8
Table 5: Cross-dataset evaluation for semantic segmentation. The mask proposal network is pre-trained on standard COCO Stuff.
Method	Source Dataset: Standard COCO Stuff	Target Dataset: ADE20K	Target Dataset: PASCAL Context
mIoU	fwIoU	pACC	mIoU	fwIoU	pACC	mIoU	fwIoU	pACC
ZSSeg [61]	40.8	49.0	62.7	19.5	48.7	60.0	50.8	64.1	75.7
POMP (Ours)	41.1	49.2	62.9	20.7	51.5	63.7	51.1	65.4	76.1
Figure 4: Qualitative results on open-vocabulary COCO-Stuff. Compared to ZSSeg, POMP correctly identifies the background category of playingfield (left) and river (right).
Table 6: Cross-dataset evaluation for object detection. The region proposal network is pre-trained on standard LVIS. POMP and Detric share the same region proposal network and training strategy.
Method	Source Dataset: Standard LVIS	Target Dataset: COCO	Target Dataset: Objects365
AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_s \)	\( AP_m \)	\( AP_l \)	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_s \)	\( AP_m \)	\( AP_l \)	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_s \)	\( AP_m \)	\( AP_l \)
ViLD* [19]	27.5	41.8	29.3	20.6	35.9	43.4	34.1	52.3	36.5	21.6	38.9	46.1	11.5	17.8	12.3	4.2	11.1	17.8
DetPro [15]	28.4	42.9	30.3	21.0	36.7	44.1	34.9	53.8	37.4	22.5	39.6	46.3	12.1	18.8	12.9	4.5	11.5	18.6
Detic [67]	36.8	50.7	38.6	26.1	46.7	51.7	38.8	56.0	41.9	25.6	42.2	50.0	15.6	22.1	16.8	6.1	15.6	23.8
POMP (Ours)	37.2	51.1	39.3	26.5	47.2	52.6	40.3	57.9	43.6	28.3	43.9	50.6	16.1	22.9	17.3	6.2	16.3	24.7
We compare POMP with state-of-the-art methods on the open-vocabulary LVIS benchmarks and report results in Table 7. POMP achieves  $\mathrm{AP}_r$  of 26.8 for object detection and 25.2 for instance segmentation. See Appendix D for qualitative results. Under the cross-dataset setting, we pre-train the visual backbone on the source dataset of standard LVIS, and evaluate the recognition ability on COCO and Object365. As shown in Table 6, compared to Detic, POMP
Table 7: Comparison with previous SOTA on LVIS dataset.  $\mathrm{AP}_r$  is the main evaluation metric for open-vocabulary object detection.
Method	Detection	Instance segmentation
APr	APc	APf	AP	APr	APc	APf	AP
ViLD [19]	16.7	26.5	34.2	27.8	16.6	24.6	30.3	25.5
DetPro [15]	20.8	27.8	32.4	28.4	19.8	25.6	28.9	25.9
PromptDet [18]	-	-	-	-	21.4	23.3	29.3	25.3
Detic [67]	26.7	36.4	40.3	36.3	24.9	32.5	35.6	32.4
POMP (Ours)	26.8	36.4	40.4	36.2	25.2	33.0	35.6	32.7
provides a gain of  $1.9\mathrm{AP}_{50}$  on COCO and  $0.8\mathrm{AP}_{50}$  on Object365, respectively.
We decouple the two components of local contrast and local correction in POMP, and conduct an ablation study to examine their individual contributions. Since removing the local contrast component will lead to prohibitive training cost, we investigate the impact of this component by varying the number of sampled classes  $K$ . As shown in Table 8, the performance of POMP improves as  $K$  increases. As discussed in § 4.3.1
Table 8: Ablation on the local contrast and local correction in POMP based on the CLIP (ViT/B-16).
Method	ImageNet-21K	Cross-dataset (10 Avg.)	Cross-domain (4 Avg.)
POMP (K=100)	24.1	65.5	59.5
POMP (K=500)	24.9	66.5	60.0
POMP (K=1000)	25.3	67.0	60.8
- local correction	25.0(-0.3)	65.8(-1.2)	59.8(-1.0)
and Table 3, the local contrast component balances accuracy and cost by adjusting  $K$ .
On the other hand, as shown in Table 8, removing the local correction component from POMP  $(K = 1000)$  results in a decline of 1.2 and 1.0 in the average accuracy of cross-dataset and cross-domain transfer, respectively. This indicates that local correction significantly improves the
Figure 5:  $\ell_{\mathrm{align}}$  and  $\ell_{\mathrm{uniform}}$  of POMP. For both measures, lower numbers are better. The color of circles and the numbers in the boxes denote the average cross-dataset accuracy over 10 datasets (higher is better).
Figure 6: Projection of image features (points), class features of POMP (intersections of solid lines and sphere) and class features of CoOp (intersections of light dash-dot lines and sphere). Each color represents a class. Class features of POMP have better alignment with centroids of the corresponding images, and are distributed with better uniformity.
generalization of the pre-trained prompt. Furthermore, we analyze the impact of the adaptive margin (5) in the local correction. We pre-train prompts on ImageNet-21K with varying  $m$  values (0, 0.5, 1, 1.5) and report the cross-dataset accuracy. Notably, our local correction method dynamically sets  $m$  to 1.5 when  $K = 319$ , and  $m$  to 1 when  $K = 1000$ . The results are shown in Table 9. When the number of sampled classes is relatively small
$(K = 319)$ , increasing the margin creates more space for potential negative classes, thereby improves cross-dataset accuracy. Conversely, for large  $K$  values (e.g., 1000), imposing a very large margin  $(m = 1.5)$  disrupts the natural class distribution and diminishes generalization ability. Overall, compared to the fixed margins [11, 54], our adaptive margin decreases as  $K$  increases, achieving optimal performance across different computing budgets (controlled by  $K$ ) and sparing the time for
extensive hyper-parameter search. See Appendix E for more ablation studies on the number of shots and prompt length.
Table 9: Ablation on the adaptive margin  $m$  in the local correction.
m	K=319	K=1000
0	65.2	65.8
0.5	65.7	66.1
1	66.2	67.0 (Ours)
1.5	66.5 (Ours)	66.4
4.5 Understanding the Pre-trained Prompt
To better understand the pre-trained prompt, we analyze the feature space of POMP through the properties of alignment and uniformity [56]. Intuitively, the image feature and its ground-truth class feature are supposed to stay closed (alignment). Besides, all the class features should be uniformly distributed to preserve maximal information and make the categories more distinguishable (uniformity). We use the alignment and uniformity loss in the vision-and-language field [45, 63] for representation probing. The alignment loss calculates the expected distance between features of an image  $\mathbf{x}$  and its ground truth class  $\mathbf{w}_y^{(\Theta)}$ :
$$
\ell_ {\text {a l i g n}} \triangleq \underset {(\mathbf {x}, y) \in \mathcal {D}} {\mathbb {E}} \left\| \mathbf {x} - \mathbf {w} _ {y} ^ {(\boldsymbol {\Theta})} \right\| ^ {2}, \tag {6}
$$
while the uniformity loss measures how well the class features  $\mathbf{w}^{(\Theta)}$  are uniformly distributed:
$$
\ell_ {\text {u n i f o r m}} \triangleq \log_ {1 \leqslant i, j \leqslant N,} \underset {i \neq j} {\mathbb {E}} \exp (- 2 \| \mathbf {w} _ {i} ^ {(\boldsymbol {\Theta})} - \mathbf {w} _ {j} ^ {(\boldsymbol {\Theta})} \| ^ {2}). \tag {7}
$$
We visualize the alignment and uniformity measures of POMP and the previous SOTA, MaPLe, in Figure 5. For both measures, lower numbers are better. The circle of POMP in the figure is located in the lower left with the lightest color, indicating relatively smaller losses and the best performance under the cross-dataset setting. Compared with the method without local correction, POMP significantly reduces the uniformity loss at only a slight expense of alignment. In other words, our pre-trained prompt not only ensures the alignment of the image and the ground-truth class, but
also disperses the class features in the representation space, thereby improving the generalization and robustness of the model. The visualization of the feature space in Figure 6 also verifies our superiority. The endpoints of the POMP class features are closer to the centroids of the image features, indicating better alignment and reduced  $\ell_{\mathrm{align}}$  loss (from 1.39 to 1.36 on Aircraft and from 1.41 to 1.36 on UCF101). Furthermore, the larger angles between the POMP class features demonstrate better feature uniformity and reduced  $\ell_{\mathrm{uniform}}$  loss (from  $-0.66$  to  $-0.81$  on Aircraft and from  $-0.95$  to  $-1.23$  on UCF101) compared to CoOp.
We present POMP to pre-train a general soft prompt on ImageNet-21K for universal visual discrimination. The learned prompt can be easily plugged into various visual recognition datasets and tasks for zero-shot inference. Experiments on open-vocabulary image classification, semantic segmentation, and object detection show that POMP surpasses previous methods by a considerable margin.
To facilitate future research, we analyze the limitations in our work and propose potential solutions. (1) We present the local contrast and use the loss within a subsampled class set as an empirical estimation for the expected contrastive loss within the full class set. However, the theoretical risk of such an estimation is urged to be investigated. (2) ImageNet-21K comprises a vast number of classes that are organized based on a semantic structure. By leveraging the hyponym and hypernym relations provided by WordNet synsets, we can derive the parent class and a list of child classes for each class. We believe that utilizing the semantic information holds the potential to further enhance performance. (3) Despite the excellent performance exhibited by our pre-trained prompt, its interpretability poses a significant challenge because the context vectors are optimized in a continuous space. We leave it as future work.
[1] Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Neural Information Processing Systems, 2019.
[2] M Saiful Bari, Aston Zhang, Shuai Zheng, Xingjian Shi, Yi Zhu, Shafiq R. Joty, and Mu Li. Spt: Semi-parametric prompt tuning for multitask prompted learning. 2022.
[3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In ECCV, 2014.
[4] Max Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. ArXiv, abs/1906.00817, 2019.
[5] Adrian Bulat and Georgios Tzimiropoulos. Language-aware soft prompting for vision & language foundation models. ArXiv, abs/2210.01115, 2022.
[6] Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1209-1218, 2016.
[7] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and-language models. In European Conference on Computer Vision, 2020.
[8] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In Neural Information Processing Systems, 2021.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[61] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. ArXiv, abs/2112.14757, 2021.
[62] Lewei Yao, Runhu Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. ArXiv, abs/2111.07783, 2021.
[63] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. ArXiv, abs/2210.07225, 2022.
[64] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122-5130, 2017.
[65] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130:2337 - 2348, 2021.
[66] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16795-16804, 2022.
[67] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krahenbuhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision, 2022.
[68] Xingyi Zhou, Vladlen Koltun, and Philipp Krahenbuhl. Probabilistic two-stage detection. ArXiv, abs/2103.07461, 2021.
[69] Benjin Zhu, Junqiang Huang, Zeming Li, Xiangyu Zhang, and Jian Sun. Eqco: Equivalent rules for self-supervised contrastive learning. ArXiv, abs/2010.01929, 2020.
A Pre-training Details
We conduct prompt pre-training on the ImageNet-21K dataset (official winter 2021 released version<sup>2</sup>). We follow the processing methods in [47], which involves cleaning invalid classes, allocating 50 images per class for a validation split, and crop-resizing all the images to 224 resolution. We conduct all the experiments on  $8 \times$  Nvidia V100 GPUs. For pre-training, the learnable vector is randomly initialized by drawing from a zero-mean Gaussian distribution with a standard deviation equal to 0.02. We use the SGD optimizer with an initial learning rate of 0.002, decayed by the cosine annealing rule. The batch size is 32, and the maximum epoch is 20.
For the mask proposal network and region proposal network pre-training, we strictly follow the settings of ZSSeg [61] and Detic [67], respectively. Specifically, we take MaskFormer [8] with ResNet-101 [23] as the mask proposal network. We use an AdamW optimizer with the initial learning rate of 1e-4, weight decay of 1e-4, a backbone multiplier of 0.1, and a poly learning rate policy with a power of 0.9. Besides, we take CenterNet2 [68] detector with ImageNet-21k pre-trained ResNet-50 [47] as the region proposal network. We use an Adam optimizer with learning rate 2e-4. Other tricks like Federated Loss, repeat factor sampling, and large scale jittering are incorporated to further improve the performance. As with Detic, we leverage both region-level and image-level supervision. We always first train a converged base-class-only model  $(4\times$  schedule) and fine-tune it with additional image-labeled data for another  $4\times$  schedule.
B Setting for Segmentation and Detection
Table 10 outlines the settings for semantic segmentation and object detection. We further introduce the settings in detail from three perspectives: backbone, data processing, and prompt.
Backbone. In general, we adopt a two-stage framework for these two tasks. At stage one, we use a pre-trained proposal network to generate a set of mask or region proposals. At stage two, we classify each proposal with the class features generated by our POMP prompt. For semantic segmentation, our POMP shares the same visual backbone as ZSSeg [61], which uses a pre-trained MaskFormer [8] with ResNet-101 [23] as default backbone to extract a set of binary masks. For object detection, our POMP shares the same visual backbone with Detic [67], which takes CenterNet2 [68] detector with ResNet-50 as its backbone, and leverages both region-level and image-level supervision.
Data Processing. We follow previous work [58, 61, 19, 67] to designate data belonging to two class sets as source data and target data, respectively. The proposal networks are pre-trained on the source data with the source class set, while conducting zero-shot evaluation on the target data with the target class set. There are two protocols for the source-target data split. The first is the open-vocabulary protocol, where the class set of one dataset is divided into two disjoint groups for the source and target data, respectively. The second protocol is the cross-dataset protocol, in which the source and target data are from two independent datasets with potentially overlapping class sets.
We introduce the details of class set splitting in the open-vocabulary protocol. COCO Stuff and Pascal VOC 2012 are the two semantic segmentation datasets using the open-vocabulary protocol. Following previous settings [58, 61], a total of 171 annotated classes in COCO Stuff are divided into 156 seen classes and 15 unseen classes. For Pascal VOC 2012, a total of 20 classes are divided into 15 seen classes and 5 unseen classes, and the provided augmented annotations are used. LVIS is the object detection dataset using the open-vocabulary protocol. The standard LVIS dataset contains object detection and instance segmentation labels for 1203 classes. The classes are divided into three groups: frequent, common, and rare, based on the number of training images. According to previous work [19], the data from the 866 frequent and common classes are considered the source data, while those from the remaining 337 rare classes are the target data in testing. We note that Detic utilizes both box-supervised data from LVIS as well as image-supervised data from ImageNet-21K that overlaps with LVIS (997 classes, 277 of which are novel classes). This allows Detic to demonstrate transfer not only from base to novel classes, but also from image-level to box-level recognition. Since Detic is the closest existing method to ours that leverages ImageNet-21K, we chose it as a strong baseline and followed its setup for fair comparison.
2https://image-net.org/
Table 10: Settings for semantic segmentation and object detection.
Task	Proposal Network	Setting	Source Data and Class Set (for proposal network pre-training)	Target Data and Class Set (for zero-shot evaluation)
Semantic Segmentation	MaskFormer (Mask Proposal Network)	Open-vocab COCO Stuff	COCO Stuff (seen)	COCO Stuff (unseen)
Open-vocab PASCAL VOC	PASCAL VOC (seen)	PASCAL VOC (unseen)
Cross-dataset	COCO Stuff	ADE20K / PASCAL Context
Object Detection	CenterNet2 detector (Region Proposal Network)	Open-vocab LVIS	LVIS (frequent+common) + ImageNet-21K (overlaps with LVIS)	LVIS (rare)
Cross-dataset	LVIS + ImageNet-21K (overlaps with LVIS)	COCO / Object365
Prompt. ZSSeg provides two kinds of prompts: hand-crafted prompts and learning-based prompts. Hand-crafted prompts include single prompt, i.e., "a sculpture of a [CLASSNAME]", as well as ImageNet prompts [42] and ViLD prompts [19], which are used for prompt ensemble and consist of 80 and 14 hard prompts, respectively. The learning-based prompt is obtained by fine-tuning a randomly initialized soft prompt on the source data. Accordingly, for a fair comparison, we conducted two sets of experiments based on whether to use the source data for prompt fine-tuning. (1) The results of ZSSeg with various hard-crafted prompts and the pre-trained POMP prompt without access to the source data can be found in Table 13 in Appendix E.3. (2) The results of ZSSeg with learning-based prompts initialized from random vectors and our pre-trained POMP prompt, both using source data for further fine-tuning, can be found in Table 4 and Table 5 in § 4.3.2. Detic has also extensively delved into intricate prompts, such as "a photo of a [CLASS] in the scene". Moreover, it has made endeavors to employ synonyms for each category. Nevertheless, its ultimate recommendation is to use a simple yet effective prompt, i.e., "a [CLASSNAME]", and all its released checkpoints are based on this prompt. We strictly adhere to Detic's best practice, the evaluation of Detic and POMP in § 4.3.3 are both conducted without any further prompt tuning on the source data.
The details of the downstream datasets for image classification, semantic segmentation, and object detection are shown in Table 11.