SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Image Classification. For cross-dataset image classification, we evaluate the performance of POMP on 10 downstream datasets, including Caltech-101 [17], Oxford-Pets [41], Stanford Cars [31], Oxford-Flowers102 [40], Food-101 [3], FGVC Aircraft [36], EuroSAT [24], SUN-397 [59], Describable Textures (DTD) [9], UCF-101 [50]. We also conduct zero-shot evaluation on 4 out-of-domain datasets derived from ImageNet [10], including ImageNetV2 [44], ImageNet-S [55], ImageNet-A [27], and ImageNet-R [26], to evaluate the domain generalization capability of our method.
Semantic Segmentation. We perform open-vocab semantic segmentation on COCO Stuff [6] and Pascal VOC 2012 [16]. Following previous notation and settings [58, 61], we split the class set into seen and unseen classes, where data for seen classes is considered the source data and data for unseen classes is considered the target data. The major measures for evaluation include mIoU and the harmonic mean IoU (hIoU) among both seen and unseen classes [61]. The hIoU is defined as:
$$
\mathrm {h I o U} = \frac {2 \times \mathrm {m I o U} _ {\text {s e e n}} \times \mathrm {m I o U} _ {\text {u n s e e n}}}{\mathrm {m I o U} _ {\text {s e e n}} + \mathrm {m I o U} _ {\text {u n s e e n}}}
$$
We also conduct cross-dataset evaluation, which takes the standard COCO Stuff dataset as the source dataset for pre-training a mask proposal network, and then conducts zero-shot inference on ADE20K [64] and PASCAL Context [38].
Object Detection. We evaluate the performance of POMP on the object detection dataset LVIS [21] under the open-vocabulary setting proposed by [19]. The source data consists of box-level data from LVIS's 866 frequent and common classes, as well as image-level data from ImageNet-21K that overlaps with LVIS. The target data for testing comprises the remaining 337 rare classes in LVIS. We take  $\mathrm{AP}_r$ , i.e., AP on rare classes, as the major measure.  $\mathrm{AP}_f$  and  $\mathrm{AP}_c$ , i.e., AP on frequent and common classes, are also reported. In the cross-dataset setting, the region proposal network is
Table 11: Datasets in our experiments.
Dataset	Classes	Train Size	Test Size	Metric
Datasets of Image Classification
Caltech-101 [17]	102	3,060	6,086	mean per-class accuracy
Oxford-IIIT Pets [41]	37	3,680	3,669	mean per-class accuracy
Stanford Cars [31]	196	8,144	8,041	accuracy
Oxford Flowers-102 [40]	102	2,040	6,149	mean per-class accuracy
Food-101 [3]	101	75,750	25,250	accuracy
FGVC Aircraft [36]	100	6,667	3,333	mean per-class accuracy
SUN-397 [59]	397	15,880	19,850	accuracy
Describable Textures (DTD) [9]	47	3,760	1,880	accuracy
EuroSAT [24]	10	10,000	5,000	accuracy
UCF-101 [50]	101	7,639	3,783	accuracy
ImageNetV2 [44]	1,000	10,000	10,000	accuracy
ImageNet-S [55]	1,000	50,889	50,889	accuracy
ImageNet-A [27]	200	7,500	7,500	accuracy
ImageNet-R [26]	200	30,000	30,000	accuracy
Datasets of Semantic Segmentation
COCO Stuff [6]	171	117K	5K	mIoU (seen/unseen), hIoU
PASCAL VOC [16]	20	11,185	1,449	mIoU (seen/unseen), hIoU
ADE20K [64]	150	20K	3K	mIoU, fwIoU, pACC
PASCAL Context [38]	59	10,103	9,637	mIoU, fwIoU, pACC
Datasets of Object Detection
LVIS [21]	1,203	100,170	19,822	APr, APc, APf, AP
COCO [34]	80	118K	5K	AP, AP50, AP75, APs, APm, APl
Object365 [48]	365	600K	38K	AP, AP50, AP75, APs, APm, APl
pre-trained on the source dataset, which includes standard LVIS and ImageNet-21K (overlapping with LVIS). It is then directly used for inference on two target datasets: COCO [34] and Object365 [48]. We use AP,  $\mathrm{AP}_{50}$ ,  $\mathrm{AP}_{75}$ ,  $\mathrm{AP}_s$ ,  $\mathrm{AP}_m$ , and  $\mathrm{AP}_l$  the evaluation metrics.
D Qualitative Results for Semantic Segmentation and Object Detection
In this section, we provide more qualitative results of our POMP for semantic segmentation and object detection. Figure 7 shows another three cases on open-vocabulary COCO-Stuff segmentation. POMP demonstrates a stronger ability than ZSSeg in the recognition of background classes. In case (1), POMP correctly identified the dirt and plant-other in the scene, instead of marking all these areas as grass. In case (2) and (3), POMP recognizes the classes of clouds and tree, respectively, while ZSSeg misclassifies them as sky-other and bush. However, POMP misses some objects of sheep located at the edge in case (2) and neglects the object of branch in case (3), indicating it still has insufficient recognition of small objects. For object detection, Figure 8 illustrates qualitative results on LVIS images. Base and novel categories are shown in purple and green, respectively. POMP identifies regions from the novel class without using the corresponding 1.2K detection annotations, demonstrating its generalization in the wild.
E More Ablation Study
E.1 Ablation on Proposal Distribution
As introduced in § 3.2, we also investigate other types of proposal distribution for local contrast and negative class sampling. The first is the frequency distribution  $Q^{(f)}$ , which samples the negative class  $i$  based on the number of training samples belonging to this class. Note that the original ImageNet-21K is class-imbalanced, i.e., the number of training samples belonging to common classes is larger than those belonging to rare classes, which can roughly reflect the long-tail distribution of object categories in nature. The frequency distribution will allow for more sampling of common classes while suppressing the exposure of rare classes in prompt tuning. Let  $M_{i}$  be the number of
(3)
ZSSeg (Baseline)
Figure 7: More qualitative results on open-vocabulary COCO-Stuff segmentation.
POMP (Ours)
Figure 8: Qualitative results on LVIS images. Base and novel categories are shown in purple and green colors respectively. We use a score threshold of 0.5 and show the most confident class for each box.
Table 12: Ablation on the sampling distribution in POMP based on CLIP (ViT/B-16) backbone.
Method	ImageNet-21K	Cross-dataset (10 Avg.)	Cross-domain (4 Avg.)
POMP (uniform distribution)	25.3	67.0	60.8
POMP (frequency distribution)	24.9 (-0.4)	66.2 (-0.8)	60.1 (-0.7)
POMP (similarity distribution)	23.6 (-1.7)	64.2 (-2.8)	59.2 (-1.6)
training samples belonging to the negative class  $i$ , the frequency distribution is defined as:
$$
Q _ {i} ^ {(f)} = \frac {M _ {i}}{\sum_ {j = 1} ^ {N} M _ {j}}. \tag {8}
$$
The second is the similarity distribution  $Q^{(s)}$ , which aims to sample more hard negative classes. Hard negative classes are those that have a higher similarity between their features and the features of the input images, and are more likely to be confused with the positive class. Accordingly, in the similarity distribution, the likelihood of a negative class being sampled increases as the similarity between its feature and the image feature increases. To achieve this, we pre-encode features of all classes represented by a hand-crafted prompt (i.e., "a photo of a [CLASSNAME]"). The feature of class  $i$  is denoted as  $\mathbf{w}_i$ . The likelihood of sampling a negative class is determined by the similarity between the class feature  $\mathbf{w}_i$  and the image feature  $\mathbf{x}$ :
$$
Q _ {i} ^ {(s)} (\mathbf {x}) = \frac {\exp \left(\mathbf {x} ^ {\top} \mathbf {w} _ {i} / \tau\right)}{\sum_ {j = 1} ^ {N} \exp \left(\mathbf {x} ^ {\top} \mathbf {w} _ {j} / \tau\right)}. \tag {9}
$$
Table 12 illustrates the performance of different proposal distributions. Compared to the uniform distribution, using the frequency distribution for sampling leads to degraded performance, particularly in cross-dataset and cross-domain settings, due to reduced sampling of rare categories. This highlights the importance of a large number of long-tail categories in the ImageNet-21K dataset for the generalization of the soft prompt. Additionally, the performance of the similarity distribution is also not as strong as that of the uniform distribution. The reason for this may be that as the soft prompt evolves, the features of hard negative classes change. However, the negative features used in (9) are obtained from the hard prompt, creating a fixed proposal distribution that is unable to adapt to these changes, potentially causing the soft prompt to converge to a local optimum. In contrast, POMP with the simple uniform distribution considers both common and rare classes, as well as easy and difficult classes, leading to the best performance for both the soft prompt and class features.
E.2 Ablation on #shot and Prompt Length
We further conduct ablation on the number of pre-training instances per class (#shot) and the prompt length to analyze their influence on the generalization ability of POMP. The left panel in Figure 9 illustrates the results of #shot. The green curve represents the average accuracy of 10 datasets under the cross-dataset evaluation, while the purple curve represents the averaged accuracy of 4 datasets under the cross-domain evaluation. Overall, the performance of POMP improves as #shot increases. We find that POMP can achieve decent cross-dataset and cross-domain accuracy even with #shot=1. This is due to the huge number of classes in ImageNet-21K. Even if there are only one instance per class, the overall amount of data (21K instances for 21K classes) is enough for training a soft prompt with only 0.012 M learnable parameters.
The right panel in the figure shows the results of the prompt length. The soft prompt of length 16 achieves  $65.0\%$  accuracy across datasets, which is lower than the soft prompt of length 4 with  $67.0\%$  cross-dataset accuracy. It indicates that the prompt with too large lengths impairs its generalization, which consistent with the findings from previous work [66, 29].
E.3 Ablation on Prompt Types for Semantic Segmentation
We perform an ablation study on prompt types for cross-dataset semantic segmentation to further demonstrate the superior generalization ability of our prompt on downstream tasks. Specifically, we
Figure 9: Ablation study on #shot and prompt length. When varying #shot, the prompt length is 4, and when varying the prompt length, #shot is 16.
Table 13: Cross-dataset evaluation for semantic segmentation. All methods share the same visual backbone with ZSSeg, but use different prompts.
Method	Source Dataset: Standard COCO Stuff	Target Dataset: ADE20K	Target Dataset: PASCAL Context
mIoU	fwIoU	mACC	pACC	mIoU	fwIoU	mACC	pACC	mIoU	fwIoU	mACC	pACC
ZSSeg (single prompt)	40.5	47.8	53.5	61.7	17.8	44.0	31.0	52.9	51.8	64.6	69.9	74.3
ZSSeg (ImageNet prompts)	40.9	48.4	54.7	62.3	17.7	46.5	31.8	57.1	52.0	64.7	70.3	75.4
ZSSeg (ViLD prompts)	40.9	48.6	54.2	62.3	20.2	49.1	33.4	60.7	51.8	63.8	69.6	73.8
ZSSeg (POMP prompt, ours)	41.2	49.0	54.7	62.6	20.6	49.3	35.0	61.7	52.4	65.3	70.6	76.4
take ZSSeg as the backbone and evaluate the performance of four types of prompts, as described in Appendix B. As shown in Table 13, ZSSeg with our POMP prompt achieves the highest performance on the three datasets. It is noteworthy that, despite using 80 hard prompts for ImageNet prompts and 14 for ViLD prompts for prompt ensemble, their performance was consistently worse than our POMP with just one soft prompt, highlighting the effectiveness of our method.