SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The author of UPL argue that self-attention allows for beneficial interaction between the two separate modalities, which leads to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features [44].
Prompts initialization We initialize textual and visual prompts from a normal distribution of mean 0 and variance 0.02. We note that we learn shallow visual prompts by modifying only the input to the image encoder. Multimodal prompts are initialized from a uniform distribution. We found that the latter was not working properly for textual and visual prompts.
Additional training settings For training, the batch size is 64.
Additional details about pseudolabels assignment It can happen that if  $K$  is too large and the unlabeled dataset is smaller than  $K \times C$ , we cannot assign  $K$  samples per class. In this case, we can reduce  $K$  accordingly. Also, we the same sample might get assigned to the pseudolabel lists of different classes. It rarely happens in our experiments cases. However, this is a characteristic of the pseudolabeling strategy proposed in [15], and it can be an object of study for future work motivated by the effectiveness of self-training.
A.2 Datasets details
We use six datasets from specialized or fine-grained domains. Here we provide a description of each of them. In Table 4, we report the details about the number of classes and data available for each dataset. For each dataset, we also show CLIP's prediction distribution over classes Figure 5.
Flowers102 [29] It is a dataset collecting images for 102 flower categories commonly occurring in the United Kingdom. For each class we have between 40 and 258 images. Figure 5a shows that CLIP's predictions are skewed toward certain classes, which are predicted more often than what we would expect according to the real class distribution on the test set.
RESICS45 [9] This is a publicly available benchmark for Remote Sensing Image Scene Classification. It collects 45 kind of scenes. Figure 5b shows that CLIP predicts more often a subset of classes.
(b) RESICS45
(c) FGVC-Aircraft
(d) MNIST
(e) EuroSAT
(f) DTD
Figure 5: For each dataset we show the distribution of CLIP's predictions over classes on the test set. The blue dots represent the true class distribution.
Multimodal prompts
Flowers102	RESICS45	FGVCAircraft
Method	SSL	UL	TRZSL	SSL	UL	TRZSL	SSL	UL	TRZSL
CLIP	63.670.00	63.400.00	54.480.00	54.460.00	17.580.00	17.860.00
UPT	68.031.29	-	61.050.04	62.841.05	-	58.790.04	11.134.98	-	15.890.07
GRIP	74.562.02	64.821.63	82.010.01	73.680.91	69.370.61	82.170.00	17.360.43	14.730.08	17.8510.30
Δ CLIP	↑10.89	↑1.15	↑18.61	↑19.2	↑14.89	↑27.71	↓0.22	↓2.85	↓0.01
Δ UPT	↑6.53	-	↑20.96	↑10.84	-	↑22.38	↑6.23	-	↑1.96
MNIST	EuroSAT	DTD
CLIP	25.100.00	20.770.00	32.880.00	30.540.00	43.240.00	43.450.00
UPT	64.443.66	-	63.590.11	68.859.92	-	60.430.04	43.712.18	-	36.910.04
GRIP	65.942.23	68.18run	73.752.93	60.384.77	61.523.04	95.520.40	54.072.25	47.370.7	63.420.00
Δ CLIP	↑40.84	↑43.08	↑52.98	↑27.5	↑28.64	↑64.98	↑10.83	↑4.13	↑19.97
Δ UPT	↑2.35	-	↑10.16	↓8.47	-	↑35.09	↑10.36	-	↑26.51
Table 5: For each learning paradigm, we compare the accuracy of GRIP with CLIP zero-shot (ViT-B/32), and UPT. Results are for SSL, UL, and TRZSL on FRAMED. We average the accuracy on 5 seeds and report the standard deviation.  $\triangle$  METHOD is the difference between the accuracy of GRIP and METHOD. We note that for UL we can not apply UPL since no labeled data is available.
FGVC-Aircraft [26] It describes the fine-grained task of categorizing aircraft. We consider the task of classifying aircrafts into 100 variants. Also for this task, CLIP assigns images to a reduced set of classes (Figure 5c).
MNIST [11] MNIST is a database of handwritten digits. The digits are size-normalized and centered in a fixed-size image. We observe that CLIP never predicts 6 out of 10 classes (Figure 5c).
EuroSAT [14] EuroSAT represents the task of categorizing satellite images of scenes. It consists of 10 classes. In Figure 5e, we show CLIP's predictions distribution over the classes.
DTD [10] DTD stands for Describable Textures Dataset. It is an evolving collection of textural images in the wild, and it is annotated relying on human-centric attributes, inspired by the perceptual properties of textures. The zero-shot CLIP predictions show the model's bias toward certain classes (Figure 5f).
A.3 Experiments
In this section, we report tables and plots that complement the results presented in Section 4.
The effect of GRIP on multimodal prompts Table 5 shows the improvements of GRIP on CLIP and Unified Prompt Tuning (UPL) [44]. Similar to the results in Table 1, GRIP consistently improves CLIP with respect to the baselines. The improvements on CLIP are by 18.2 in semi-supervised learning, 14.8 in unsupervised learning, and 30.7 in transductive zero-shot learning. While GRIP outperforms UPL by 4.7 in semi-supervised learning, and 19.5 in transductive zero-shot learning.
Comparison across iterative strategies In Table 6, we report a comparison between FPL and the iterative strategies (IFPL and GRIP) on MNIST, EuroSAT, and FGVC-Aircraft. Results on the other tasks can be found in the main body of the paper Section 4.1. While GRIP largely and consistently outperforms FPL by on average 16.7 points in accuracy, IFPL is not robust and it leads to performances that are inferior to FPL by on average 4.4 points in accuracy.
The evolving accuracy of dynamic pseudolabels Figure 6 represents the evolution of pseudolabels accuracy during training for all datasets, but Flowers102 and RESICS45 presented in Figure 3. We observe that the accuracy of the pseudolabels characterizes the overall performance of the models reported in Table 6. For instance, IFPL for EuroSAT in the TRZSL setting is highly variable, explaining the low average accuracy of the model on the test set (Table 6). Similarly, for MNIST in the TRZSL we observe that after the first iteration, the pseudolabels get very noisy. When we increase the amount of pseudolabeled data, the accuracy of CLIP is not necessarily constant. This is because as we increase K, we are effectively selecting pseudolabels with lower similarities to the
Textual prompts
MNIST	EuroSAT	FGVCAircraft
Method	SSL	UL	TRZSL	SSL	UL	TRZSL	SSL	UL	TRZSL
FPL	66.061.10	40.032.63	9.7319.45	62.051.64	48.961.49	53.7026.87	20.020.77	16.620.67	17.550.37
IFPL	59.143.43	28.942.05	0.000.00	61.281.59	56.463.26	14.3628.71	18.000.35	13.800.67	21.720.77
GRIP	71.783.59	67.882.76	74.060.29	58.662.64	57.211.77	92.330.69	16.980.82	15.220.71	26.080.25
Δ IFPL	↓6.92	↓11.09	↓9.73	↓0.77	↑7.50	↓39.34	↓2.02	↓2.82	↑4.17
Δ GRIP	↑5.72	↑27.85	↑64.33	↓3.39	↑8.25	↑38.63	↓3.04	↓1.40	↑8.53
Visual prompts
FPL	42.8416.80	39.626.53	31.8217.53	52.472.53	48.793.69	68.6814.74	20.140.26	18.280.33	16.280.45
IFPL	52.918.99	37.176.27	38.384.21	57.856.52	32.5210.00	48.1311.13	18.770.48	16.360.37	19.290.36
GRIP	69.665.51	68.041.11	69.541.31	63.483.09	63.683.42	96.970.77	19.430.50	17.510.61	26.420.30
Δ IFPL	↑10.07	↓2.45	↑6.56	↑5.38	↓16.27	↓20.55	↓1.37	↓1.92	↑3.01
Δ GRIP	↑26.82	↑28.42	↑37.72	↑11.01	↑14.89	↑28.29	↓0.71	↓0.77	↑10.14
Table 6: For each learning paradigm, we compare FPL, IFPL, and GRIP on MNIST, EuroSAT, and FGVCAircraft. We average across 5 runs and report the standard deviation.  $\Delta$  METHOD is the difference between the accuracy of FPL and METHOD.
(a) DTD
(b) EuroSAT
(c) MNIST
(d) FGVC-Aircraft
Figure 6: We plot the evolution of dynamic-pseudolabels accuracy during training. The rows refer to SSL, UL, and TRZSL, in order. IFPL refers to the top x-axis, while CLIP and GRIP to the bottom.
classes, resulting in a reduction in their accuracy, as shown in the plot. This observation aligns with previous findings in [15].
GRIP performance on transductive zero-shot learning We show how the effectiveness of GRIP is consistent over the three random splits of seen and unseen classes which we randomly generated. The splits are reported in Table 9. Table 8 gathers the accuracy of seen and unseen classes, along with the harmonic mean for all three splits using textual prompts. Beyond the consistent improvement induced by GRIP training strategy, we observe that the accuracy of GRIP on the seen classes is often lower than the accuracy of CoOp on the same set of classes. During training, for large lambda, the loss component of unlabeled data (unseen classes) is the first to decrease, while the loss on the seen classes reduces later. Thus, we hypothesize that extra training steps might be needed to complete the learning on the labeled data.
A.4 The Robin Hood effect
The Robin Hood effect on all tasks For each dataset, we provide the per-class accuracy distribution of GRIP compared with CLIP, Figure 8. The Robin Hood effect characterizes all the tasks. We observe that for GRIP the increase in overall accuracy corresponds to consistent improvements in the predictions of initially poor classes. By comparing Figure 7 with Figure 8, we see that GRIP reinforces the Robin Hood effect already visible when using FPL in certain cases.
The importance of good quality pseudolabels to mitigate the Matthew effect in SSL In the SSL setting, we train a logistic regression on top of the visual feature extracted by CLIP's image encoder (ViT-B/32). In Figure 9, we show the per-class accuracy of the final model trained by combining labeled data with either pseudolabels assigned with the conventional scheme (threshold at .95) or 16 CLIP-generated pseudolabels. We compare the two distribution with the per-class accuracy of the model trained solely on the few labeled examples per class (2 instances).
The different impact of prompt tuning and linear probing on the Robin Hood effect We investigate if there is any difference in the Robin Hood effect when adapting CLIP via prompt tuning or linear probing. We train both relying on the iterative training strategy that grows the set of pseudolabels at each iteration by using the top-  $K$  scheme (Section 3). We consider the UL setting.
Among the set of target classes, we distinguish between poor and rich classes. A class is poor, if CLIP's accuracy on that class is lower than its overall accuracy on the task. Otherwise, the class is considered rich. Table 7 reports the accuracy of the two approaches, and the accuracy on the poor and rich classes, while highlighting the average effect with respect to CLIP. Training with prompt tuning retains more knowledge of the rich classes than linear probing. Prompt tuning reduces the accuracy on the rich classes by on average 0.3 points, while linear probing has an average deterioration of 9.4. The reduction of accuracy of rich classes is on average 30 times larger than the reduction observed using prompts. Overall, GRIP works better than linear probing. We note that the lower accuracy of linear probing is characterized by a worse ability to correctly predict the rich classes, i.e., "rich get poorer." This is surprising, as we would have expected the errors to concentrate on the poor classes compared to CLIP.
(b) FGVC-Aircraft
(d) EuroSAT
Figure 7: Per-class accuracy of FPL compared to CLIP's per-class accuracy on Flowers102, FGVC-Aircraft, MNIST, EuroSAT, DTD. X-axis is the ranked class index, while the y-axis is the accuracy.
(e) DTD
Figure 8: Per-class accuracy of GRIP compared to CLIP's per-class accuracy on Flowers102, FGVC-Aircraft, MNIST, EuroSAT, and DTD. X-axis is the ranked class index, while the y-axis is the accuracy.
(d) EuroSAT
(e) DTD
(c) FGVCA
(d) MNIST
Figure 9: Per-class accuracy of a logistic classifier using conventional pseudolabels (first row) and CLIP-based pseudolabels (second row). The solid orange line represents the per-class accuracy of a logistic regression trained on 2-shots per class. X-axis is the ranked class index, while the y-axis is the accuracy. We present results for Flowers102, RESICS45, FGVC-Aircraft, MNIST, EuroSAT, and DTD, in order.
(e) EuroSAT
(f) DTD
Flowers102	RESICS45	FGVC-Aircraft	MNIST	EuroSAT	DTD	Avg. Δ
Linear probe (LP)	41.01	58.79	61.94	50.52	51.37	10.17	-
GRIP	46.09	70.55	69.84	57.21	67.88	15.22	-
Rich CLIP	67.81	75.47	85.16	65.26	65.14	45.93	-
Rich LP	52.87	69.01	79.55	67.53	50.34	29.12	-
Rich GRIP	56.05	78.81	86.40	71.73	77.84	31.95	-
Δ LP	↓14.92	↓6.47	↓5.61	↑2.26	↓14.79	↓16.81	↓9.39
Δ GRIP	↓11.76	↑3.33	↑1.24	↑6.46	↑12.70	↓13.98	↓0.33
Poor CLIP	25.63	35.60	27.98	11.10	3.18	5.35	-
Poor LP	26.50	42.77	36.25	28.34	56.76	4.77	-
Poor GRIP	35.03	56.85	42.82	39.88	65.08	6.31	-
Δ LP	↑0.87	↑7.18	↑8.27	↑17.24	↑53.58	↓0.58	↑14.43
Δ GRIP	↑9.4	↑21.26	↑14.84	↑28.78	↑61.9	↑0.96	↑22.86
Table 7: For each task we report the overall accuracy of linear probing (LP) and GRIP textual along with the accuracy on poor and rich classes.  $\Delta$  METHOD is the difference between the accuracy of CLIP and METHOD. For an overall evaluation of the difference between linear probing and prompt tuning, we report the average difference of LP and GRIP with respect to CLIP on poor and rich classes.
Split 1
Flowers102	RESICS45	FGVCAircraft
Method	S	U	H	S	U	H	S	U	H
CLIP	64.260.00	62.560.00	63.40.00	54.850.00	54.080.00	54.460.00	16.270.00	19.790.00	17.860.00
CoOp	91.520.36	48.352.96	63.222.60	84.661.01	50.733.28	63.372.23	34.181.56	16.283.69	21.703.45
GRIP	90.310.51	82.571.26	86.260.81	82.680.47	79.530.72	81.070.37	22.250.07	31.510.59	26.080.25
Δ CLIP	↑26.05	↑20.01	↑22.86	↑27.83	↑25.45	↑26.61	↑5.98	↑11.72	↑8.22
Δ CoOp	↓1.21	↑34.22	↑23.04	↓1.98	↑28.8	↑17.7	↓11.93	↑15.23	↑4.38
MNIST	EuroSAT	DTD
CLIP	31.740.00	15.430.00	20.770.00	22.330.00	48.30.00	30.540.00	42.50.00	44.440.00	43.450.00
CoOp	94.685.64	15.437.75	21.1512.18	82.918.81	46.029.23	58.645.86	69.671.17	34.813.44	46.32.92
GRIP	95.130.11	60.630.44	74.060.29	91.750.53	92.910.91	92.330.70	68.260.69	62.611.87	65.301.03
Δ CLIP	↑63.39	↑45.2	↑53.29	↑69.42	↑44.61	↑61.79	↑25.76	↑18.17	↑21.85
Δ CoOp	↑0.45	↑45.2	↑52.91	↑8.84	↑46.89	↑33.69	↓1.41	↑27.8	↑19.00
Split 2
Flowers102	RESICS45	FGVCAircraft
Method	S	U	H	S	U	H	S	U	H
CLIP	65.380.00	60.640.00	62.920.00	59.50.00	47.060.00	52.550.00	17.300.00	18.120.00	17.700.00
CoOp	91.81.32	47.753.86	62.773.31	86.541.92	48.003.01	61.702.17	33.594.12	19.571.37	24.630.63
GRIP	88.840.75	70.932.08	78.861.26	84.470.41	84.091.01	84.280.73	22.130.24	28.320.33	24.840.05
Δ CLIP	↑23.46	↑10.29	↑15.94	↑27.83	↑25.45	↑26.61	↑4.83	↑10.20	↑7.14
Δ CoOp	↓2.96	↑23.18	↑16.09	↓2.07	↑36.09	↑22.58	↓11.46	↑8.75	↑0.21
MNIST	EuroSAT	DTD
CLIP	15.990.00	39.180.00	22.710.00	32.470.00	33.100.00	32.780.00	45.430.00	39.720.00	42.390.00
CoOp	90.613.02	18.779.12	30.2912.38	86.433.23	47.1611.17	60.538.42	70.41.99	32.534.58	44.424.63
GRIP	95.71	97.50	96.59	91.080.02	92.020.98	91.550.47	66.690.53	56.191.18	60.990.69
Δ CLIP	↑85.12	↑50.76	↑79.32	↑58.61	↑58.92	↑58.77	↑21.26	↑16.47	↑18.6
Δ CoOp	↑6.11	↑71.20	↑57.19	↑4.65	↑44.86	↑31.02	↓3.71	↑23.66	↑16.57
Split 3
Flowers102	RESICS45	FGVCAircraft
Method	S	U	H	S	U	H	S	U	H
CLIP	68.290.00	57.250.00	62.280.00	56.020.00	52.320.00	54.100.00	17.550.00	17.710.00	17.630.00
CoOp	91.520.35	48.352.95	63.222.60	87.612.17	43.644.97	58.144.12	37.771.92	16.463.23	22.773.09
GRIP	90.090.53	69.002.44	78.131.71	85.190.15	75.583.17	80.071.79	22.070.23	28.720.76	24.950.20
Δ CLIP	↑ 21.8	↑ 11.75	↑ 15.85	↑ 29.17	↑ 23.26	↑ 25.97	↑ 4.52	↑ 11.01	↑ 7.32
Δ CoOp	↓ 1.43	↑ 20.65	↑ 14.91	↓ 2.42	↑ 31.94	↑ 21.93	↓ 15.70	↑ 12.26	↑ 2.18
MNIST	EuroSAT	DTD
CLIP	10.590.00	46.740.00	17.270.00	41.470.00	19.600.00	26.620.00	45.520.00	39.580.00	42.340.00
CoOp	89.68.08	26.312.88	39.416.61	79.339.37	43.3812.49	55.068.62	70.533.11	24.945.37	36.635.57
GRIP	95.8	96.06	95.93	90.570.13	94.251.10	92.370.60	67.280.74	58.942.78	62.811.75
Δ CLIP	↑ 79.81	↑ 56.88	↑ 73.22	↑ 49.1	↑ 74.65	↑ 65.75	↑ 21.76	↑ 19.36	↑ 20.47
Δ CoOp	↑ 5.20	↑ 77.29	↑ 65.64	↑ 11.24	↑ 50.87	↑ 37.31	↓ 3.25	↑ 34.00	↑ 26.18
Table 8: In the TRZSL settings, for each dataset and split, we compare the accuracy of GRIP textual with CLIP zero-shot (ViT-B/32), and CoOp. Results show the accuracy on seen  $(S)$  and unseen classes  $(U)$ , and the harmonic mean  $(H)$ . We average the accuracy on 5 seeds and report the standard deviation.  $\Delta$  METHOD is the difference between the accuracy of GRIP and METHOD.
Table 9: For each dataset, we report the class names of seen and unseen classes in each of the splits used for TRZSL.
Textual prompts
Flowers102	RESICS45	DTD
Method	SSL	UL	TRZSL	SSL	UL	TRZSL	SSL	UL	TRZSL
CLIP (ViT-B/32)	63.67	63.67	63.40	54.48	54.48	54.46	43.24	43.24	43.45
GRIP (ViT-B/32)	83.60	69.84	86.26	74.11	70.55	81.07	56.07	46.09	65.30
CLIP (ViT-L/14)	73.98	73.98	73.05	62.67	62.67	62.13	52.45	52.45	51.61
GRIP (ViT-L/14)	94.21	82.33	96.18	81.53	76.86	86.88	60.91	54.40	64.92
Visual prompts
CLIP (ViT-B/32)	63.67	63.67	63.40	54.48	54.48	54.46	43.24	43.24	43.45
GRIP (ViT-B/32)	67.95	63.09	77.18	71.22	68.43	82.19	54.57	50.51	62.78
CLIP (ViT-L/14)	73.98	73.98	73.05	62.67	62.67	62.13	52.45	52.45	51.61
GRIP (ViT-L/14)	78.68	73.50	85.85	77.53	76.00	86.63	55.72	54.27	62.74
Table 10: Performance of CLIP and GRIP with different backbones on Flowers102, RESICS45, and DTD, for all the learning settings SSL, UL, TRZSL.