SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	Color	Shape	Material	Cooccur
Spearman ρ	Acc@1	Spearman ρ	Acc@1	Spearman ρ	Acc@1	Spearman ρ
BERTb	47.5 ± 21.6	41.8	48.2 ± 12.0	64.3	41.9 ± 15.4	55.3	6.1 ± 4.0
Oscarb	50.0 ± 19.8	59.8	52.7 ± 10.0	89.3	46.5 ± 13.7	74.6	10.1 ± 7.2
Distilled	53.7 ± 21.3	57.7	51.4 ± 11.1	74.3	46.0 ± 13.6	74.6	10.4 ± 7.8
RoBERTa,b	44.8 ± 19.8	41.6	45.4 ± 12.4	69.3	33.0 ± 15.5	39.1	1.1 ± 1.4
ALBERTb	20.2 ± 24.8	13.4	29.8 ± 15.7	13.6	25.0 ± 17.9	27.8	6.6 ± 5.1
Vokenization	47.6 ± 20.9	51.6	49.8 ± 13.1	72.9	39.4 ± 16.0	52.5	6.0 ± 3.7
VisualBERT	52.4 ± 19.8	65.3	48.7 ± 12.9	66.4	43.4 ± 15.5	59.5	10.7 ± 8.1
CaptionBERT	55.8 ± 20.6	70.0	51.3 ± 11.8	91.4	42.6 ± 15.4	54.6	10.2 ± 7.5
Per-Subject Comparison Fig. 6 and Fig. 7 show how the Spearman correlations of 10 individual subjects improve after soft prompt tuning and after multimodal pretraining. Consistent improvement can be seen in color, material, and cooccurrence. Although we report average Spearman correlations in Table 3 and there are large standard deviations, here we show that when improvement is observed collectively, it is also consistent across subjects. With shape, the improvement is less obvious (45.9 to 50.4 for prompt tuning and 49.2 to 50.4 for multimodal pretraining).
A.3 Error Analysis
Data The three subjects with the highest and lowest Spearman correlation are shown in Fig. 8 and Fig. 9.
Wikipedia Table 10 shows the number of (noun, attribute) pairs of the three relation types in Wikipedia. Shape has fewer occurrences than material and color.
Table 9: Spearman correlation and top-1 accuracy (both  $\times 100$ ) of zero shot probing. This is the "best template" case discussed in Section 4.1.
Color	Shape	Material
Total	331480	195921	307879
Avg 12	27623.3	16326.8	24634.7
Table 10: First row is the total number of occurrences of (noun, attribute) pairs for relations shape, material, and color in Wikipedia. Second row is the average number of occurrences across the top 12 attributes for each relation. Shape has the fewest number of occurrences.
Model Table 11 shows the errors made by BERT and Oscar in the "average template" mode before prompt tuning. Overall, subjects with low correlation are those that are less often reported in Visual Genome as well as in textual data.
A.4 Resources
BERT, RoBERTa, ALBERT We use the Huggingface implementations of BERT, RoBERTa, and
Oscar See the GitHub repository for the code and pretrained Oscar: https://github.com/microsoft/Oscar.
CLIP We use the CLIP model released by OpenAI: https://github.com/openai/CLIP.
Vokenization See the GitHub repository for the pretrained model: https://github.com/airsplay/vokenization.
Figure 6: Spearman correlation of 10 subjects for each relation type before and after soft prompt tuning, with Oscar (base). Almost all individual subject has increased correlation after prompt tuning, except in relation shape.
Figure 7: Spearman correlation of 10 subjects for each relation type with BERT (base) and Oscar (base), after soft prompt tuning. Almost all individual subject has higher correlation with Oscar than with BERT, except in relation shape.
Relation	High Corr Subjs	Low Corr Subjs
BERTb	Oscarb	BERTb	Oscarb
Color	lace, jacket, design	balloon, jacket, apple	flush, water faucet, muffler	hinge, leg, slack
Shape	mirror, vase, container	chair, pizza, vase	connector, log, knot	banana, toast, phone
Material	wall, tray, board	fence, wall, shelf	sheep, fabric, patch	elephant, rug, patch
Table 11: Three subjects each with high and low correlations for relations color, shape, and material.