SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method	EuroSAT	OxfordPets	DTD	Caltech101	FGVCAircraft	UCF101	Flowers102	Average
RN50	Random	0.323 ± 0.069	0.292 ± 0.023	0.270 ± 0.008	0.362 ± 0.010	0.062 ± 0.005	0.175 ± 0.041	0.151 ± 0.011	0.234
USL-I	0.389 ± 0.042	0.234 ± 0.041	0.257 ± 0.017	0.349 ± 0.013	0.070 ± 0.007	0.188 ± 0.014	0.153 ± 0.026	0.234
USL-M	0.386 ± 0.047	0.298 ± 0.032	0.269 ± 0.043	0.340 ± 0.021	0.075 ± 0.012	0.188 ± 0.009	0.133 ± 0.031	0.241
Ours f(·)	0.398 ± 0.010	0.307 ± 0.007	0.342 ± 0.010	0.399 ± 0.005	0.084 ± 0.001	0.245 ± 0.006	0.187 ± 0.007	0.280
RN101	Random	0.291 ± 0.049	0.251 ± 0.028	0.216 ± 0.031	0.319 ± 0.016	0.058 ± 0.012	0.142 ± 0.028	0.136 ± 0.018	0.202
USL-I	0.325 ± 0.047	0.225 ± 0.047	0.205 ± 0.027	0.291 ± 0.012	0.066 ± 0.011	0.144 ± 0.014	0.124 ± 0.028	0.197
USL-M	0.319 ± 0.062	0.298 ± 0.047	0.213 ± 0.031	0.294 ± 0.018	0.064 ± 0.018	0.149 ± 0.017	0.109 ± 0.026	0.207
Ours f(·)	0.325 ± 0.000	0.294 ± 0.005	0.305 ± 0.010	0.345 ± 0.004	0.077 ± 0.000	0.224 ± 0.005	0.159 ± 0.005	0.247
ViTB32	Random	0.438 ± 0.066	0.488 ± 0.070	0.400 ± 0.024	0.427 ± 0.019	0.085 ± 0.007	0.201 ± 0.011	0.270 ± 0.040	0.330
USL-I	0.496 ± 0.039	0.438 ± 0.030	0.377 ± 0.017	0.393 ± 0.017	0.097 ± 0.010	0.223 ± 0.023	0.263 ± 0.028	0.327
USL-M	0.552 ± 0.054	0.524 ± 0.034	0.409 ± 0.016	0.390 ± 0.021	0.102 ± 0.020	0.223 ± 0.011	0.237 ± 0.035	0.348
Ours f(·)	0.541 ± 0.006	0.488 ± 0.019	0.427 ± 0.001	0.448 ± 0.004	0.104 ± 0.004	0.274 ± 0.003	0.309 ± 0.001	0.370
ViTH14	Random	0.505 ± 0.110	0.560 ± 0.029	0.421 ± 0.024	0.425 ± 0.009	0.152 ± 0.012	0.282 ± 0.027	0.290 ± 0.038	0.376
USL-I	0.585 ± 0.052	0.518 ± 0.036	0.408 ± 0.017	0.369 ± 0.015	0.147 ± 0.011	0.301 ± 0.026	0.313 ± 0.040	0.377
USL-M	0.647 ± 0.057	0.629 ± 0.032	0.435 ± 0.020	0.372 ± 0.025	0.160 ± 0.027	0.292 ± 0.008	0.294 ± 0.039	0.404
Ours f(·)	0.615 ± 0.028	0.600 ± 0.018	0.446 ± 0.005	0.435 ± 0.006	0.148 ± 0.009	0.323 ± 0.008	0.327 ± 0.001	0.413
ViTG14	Random	0.567 ± 0.062	0.592 ± 0.039	0.440 ± 0.026	0.413 ± 0.008	0.165 ± 0.026	0.279 ± 0.023	0.341 ± 0.023	0.400
USL-I	0.578 ± 0.044	0.546 ± 0.032	0.411 ± 0.017	0.372 ± 0.019	0.157 ± 0.010	0.312 ± 0.025	0.336 ± 0.037	0.387
USL-M	0.627 ± 0.052	0.658 ± 0.031	0.436 ± 0.017	0.371 ± 0.023	0.171 ± 0.023	0.303 ± 0.004	0.316 ± 0.034	0.412
Ours f(·)	0.614 ± 0.007	0.636 ± 0.014	0.445 ± 0.007	0.434 ± 0.005	0.165 ± 0.011	0.326 ± 0.009	0.342 ± 0.002	0.423
Table 7: Impact of Annotation Budget. Accuracy (\%) of linear probe results for CLIP trained with data pre-selected by random, USL-I, USL-M, and our method after increasing  $50\%$  annotation budget.
Dataset	Classes	Train	Val	Test	Hand-crafted prompt
EuroSAT	10	13,500	5,400	8,100	“a centered satellite photo of [CLASS].”
OxfordPets	37	2,944	736	3,669	“a photo of a [CLASS], a type of pet.”
DTD	47	2,820	1,128	1,692	“[CLASS] texture.”
Caltech101	100	4,128	1,649	2,465	“a photo of a [CLASS].”
FGVCAircraft	100	3,334	3,333	3,333	“a photo of a [CLASS], a type of aircraft.”
UCF101	101	7,639	1,898	3,783	“a photo of a person doing [CLASS].”
Flowers102	102	4,093	1,633	2,463	“a photo of a [CLASS], a type of flower.”
Table 8: Datasets Statistics. The detailed statistics of the 7 datasets and the hand-crafted prompts that are used for BLIP-2 zero-shot learning.
D Visualization on Selected Instances
Figure 4(c) and (d) present the visualizations of the least-40 and top-40 instances, respectively, that were selected using our method on the EuroSAT dataset. We compare these selections with the top 40 sampled images chosen through random selection and USL. In order to facilitate understanding, we have organized the images into different rows based on their ground truth labels. Note that it is important to have a balanced selection that covers all semantic classes. However, the randomly selected instances often exhibit significant imbalances. This means that we may end up with, for example, 9 instances from Class 7 while completely missing any images from Class 5. Such imbalances are quantified using the KL divergence, which measures the distance between the current sample distribution and a uniform distribution. In this case, the KL divergence is 0.241. In contrast, our top selected instances not only provide representation from each class but also offer diversity among the selected instances. The KL divergence for our top selections is only 0.047, indicating that they are both representative and diverse. On the other hand, the 40 instances that are least likely to be selected primarily consist of outliers. These outliers have the potential to mislead downstream classifiers and introduce noise into the learning process. For instance, the first image in class 0 (AnnualCrop) may appear strikingly similar to images in Class 1 (Forest) and Class 9 (SeaLake), which can be confusing for the classification task.
Figure 4: Visualizations of Selected Instances from EuroSAT. Our selection ensures balance and representativeness. In comparison, random selection can lead to imbalance and the potential omission of certain classes, particularly in low-budget settings. KLDiv represents the Kullback-Leibler divergence score between the sampling distribution and the normal distribution (stratified sampling). Classes 0-9 correspond to AnnualCrop, Forest, HerbaceousVegetation, Highway, Industrial, Pasture, PermanentCrop, Residential, River, and SeaLake, respectively.