Method	EuroSAT	OxfordPets	DTD	Caltech101	FGVCAircraft	UCF101	Flowers102	Average
RN50	Random	0.221 ± 0.055	0.193 ± 0.032	0.202 ± 0.016	0.336 ± 0.021	0.057 ± 0.009	0.113 ± 0.012	0.121 ± 0.015	0.178
USL-I	0.309 ± 0.052	0.161 ± 0.020	0.188 ± 0.028	0.297 ± 0.008	0.048 ± 0.008	0.133 ± 0.013	0.104 ± 0.043	0.177
USL-M	0.289 ± 0.044	0.191 ± 0.038	0.185 ± 0.030	0.286 ± 0.008	0.064 ± 0.010	0.108 ± 0.027	0.130 ± 0.026	0.179
Ours f(·)	0.456 ± 0.040	0.229 ± 0.002	0.283 ± 0.011	0.324 ± 0.001	0.066 ± 0.001	0.184 ± 0.010	0.137 ± 0.003	0.240
Ours gI(·)	0.469 ± 0.038	0.214 ± 0.007	0.282 ± 0.007	0.324 ± 0.001	0.074 ± 0.012	0.179 ± 0.009	0.136 ± 0.003	0.240
Ours gC(·)	0.487 ± 0.001	0.204 ± 0.013	0.283 ± 0.007	0.353 ± 0.002	0.076 ± 0.004	0.205 ± 0.006	0.130 ± 0.001	0.248
RN101	Random	0.217 ± 0.062	0.193 ± 0.021	0.139 ± 0.026	0.274 ± 0.023	0.042 ± 0.013	0.093 ± 0.014	0.116 ± 0.040	0.153
USL-I	0.257 ± 0.038	0.149 ± 0.019	0.145 ± 0.049	0.263 ± 0.012	0.042 ± 0.006	0.101 ± 0.017	0.090 ± 0.042	0.150
USL-M	0.209 ± 0.030	0.190 ± 0.052	0.144 ± 0.035	0.257 ± 0.010	0.058 ± 0.013	0.072 ± 0.037	0.111 ± 0.026	0.149
Ours f(·)	0.350 ± 0.051	0.211 ± 0.004	0.250 ± 0.017	0.298 ± 0.001	0.056 ± 0.003	0.146 ± 0.013	0.133 ± 0.005	0.206
Ours gI(·)	0.408 ± 0.024	0.197 ± 0.011	0.245 ± 0.018	0.294 ± 0.006	0.070 ± 0.012	0.145 ± 0.010	0.133 ± 0.002	0.213
Ours gC(·)	0.423 ± 0.007	0.177 ± 0.012	0.260 ± 0.010	0.312 ± 0.002	0.067 ± 0.006	0.190 ± 0.015	0.120 ± 0.001	0.221
ViTB32	Random	0.332 ± 0.044	0.340 ± 0.024	0.286 ± 0.049	0.371 ± 0.014	0.071 ± 0.015	0.150 ± 0.013	0.175 ± 0.024	0.246
USL-I	0.422 ± 0.096	0.299 ± 0.038	0.275 ± 0.028	0.318 ± 0.006	0.074 ± 0.006	0.164 ± 0.020	0.179 ± 0.036	0.247
USL-M	0.412 ± 0.077	0.365 ± 0.033	0.290 ± 0.034	0.310 ± 0.010	0.091 ± 0.008	0.132 ± 0.035	0.188 ± 0.025	0.255
Ours f(·)	0.525 ± 0.010	0.439 ± 0.011	0.372 ± 0.002	0.352 ± 0.002	0.089 ± 0.001	0.214 ± 0.011	0.188 ± 0.008	0.311
Ours gI(·)	0.557 ± 0.006	0.429 ± 0.019	0.379 ± 0.002	0.355 ± 0.005	0.094 ± 0.013	0.207 ± 0.013	0.186 ± 0.006	0.315
Ours gC(·)	0.584 ± 0.013	0.380 ± 0.020	0.385 ± 0.007	0.392 ± 0.002	0.098 ± 0.006	0.234 ± 0.010	0.217 ± 0.003	0.327
ViTH14	Random	0.482 ± 0.099	0.404 ± 0.042	0.278 ± 0.034	0.330 ± 0.016	0.118 ± 0.025	0.174 ± 0.024	0.229 ± 0.031	0.288
USL-I	0.504 ± 0.103	0.359 ± 0.035	0.284 ± 0.035	0.298 ± 0.016	0.108 ± 0.010	0.206 ± 0.022	0.227 ± 0.024	0.284
USL-M	0.505 ± 0.070	0.434 ± 0.029	0.304 ± 0.037	0.301 ± 0.015	0.128 ± 0.013	0.180 ± 0.036	0.221 ± 0.018	0.296
Ours f(·)	0.577 ± 0.011	0.567 ± 0.005	0.392 ± 0.003	0.335 ± 0.001	0.116 ± 0.001	0.253 ± 0.017	0.206 ± 0.000	0.349
Ours gI(·)	0.596 ± 0.007	0.548 ± 0.025	0.394 ± 0.003	0.332 ± 0.003	0.131 ± 0.020	0.247 ± 0.005	0.203 ± 0.001	0.350
Ours gC(·)	0.634 ± 0.014	0.477 ± 0.021	0.403 ± 0.009	0.371 ± 0.005	0.143 ± 0.003	0.287 ± 0.010	0.241 ± 0.001	0.365
ViTG14	Random	0.402 ± 0.045	0.421 ± 0.027	0.305 ± 0.033	0.334 ± 0.022	0.109 ± 0.014	0.198 ± 0.035	0.216 ± 0.032	0.284
USL-I	0.499 ± 0.106	0.383 ± 0.029	0.285 ± 0.033	0.302 ± 0.013	0.113 ± 0.014	0.208 ± 0.019	0.243 ± 0.026	0.290
USL-M	0.481 ± 0.087	0.461 ± 0.026	0.305 ± 0.038	0.298 ± 0.017	0.133 ± 0.013	0.184 ± 0.040	0.238 ± 0.024	0.300
Ours f(·)	0.560 ± 0.024	0.582 ± 0.008	0.389 ± 0.004	0.339 ± 0.002	0.130 ± 0.005	0.254 ± 0.014	0.224 ± 0.002	0.354
Ours gI(·)	0.598 ± 0.004	0.566 ± 0.023	0.388 ± 0.006	0.336 ± 0.001	0.145 ± 0.021	0.252 ± 0.003	0.216 ± 0.004	0.357
Ours gC(·)	0.609 ± 0.019	0.503 ± 0.017	0.402 ± 0.005	0.376 ± 0.004	0.161 ± 0.005	0.289 ± 0.015	0.258 ± 0.002	0.371
Zero Shot BLIP-2	0.111 (0.100)	0.081 (0.027)	0.123 (0.021)	0.379 (0.010)	0.113 (0.010)	0.070 (0.010)	0.114 (0.010)	0.141
Table 1: Accuracy (\%) of linear probe results for CLIP trained with data pre-selected by random, USL-I, USL-M, and our method. Evaluation is carried out using five different versions of the CLIP vision encoder. The accuracy (\%) of zero-shot performance with BLIP-2 is displayed in the bottom row, with the numbers in brackets representing the random guessing baseline. Our UP-DP consistently outperforms other baselines by an average accuracy around  $7\%$ .