SlowGuess/NeurIPS_2023_OCR huggingface dataset:
performed comparably to the None encoder baseline. W2V-P also fails, where the fixed random context tokens mislead the image features. In contrast, the CLIP outperforms both of them. The fixed W2V embedding vectors struggled to correlate well with target image features. Moreover, BERT surpassed W2V and W2V-P due to the generalizability of a stronger pre-trained model. However, BERT still cannot achieve better performances than CLIP. Furthermore, BERT-ALIGN performs competitively and even better than CLIP, which is attributed to its larger training set with more noisy image-text pairs. Consequently, from the results in Table 12 and this paper, we conclude that LICO works better with those vision-language pre-trained text encoders. Pure text encoders like W2V, even the pre-trained BERT with frozen parameters, are inferior in image-text alignment in LICO because the pre-trained parameters are not sensitive to visual features. This deficiency may be addressed by utilizing some transfer learning and domain adaptation tricks.
Table 12: Comparison of different text encoders on CIFAR-10.
Encoder	CLIP	W2V	W2V-P	BERT	BERT-ALIGN	None
Full	95.8	95.6	94.9	95.7	95.8	95.6
4000	81.5	81.0	80.2	81.3	81.7	80.9
G Frozen Parameters of Prompts
To evaluate the model performance with frozen parameters, we then conducted experiments on CIFAR-10 and ImageNet under two settings of frozen parameters: (1) frozen random initialization (Random) and (2) fixed form of 'a photo of a [CLS]' (Fixed). The insertion and deletion values are obtained by Grad-CAM + LICO.
In Table 13, we can see that the frozen random parameters cannot enable the models to achieve higher performances of accuracy, insertion, and deletion; the reason is that the well-trained CLIP text encoder is capable of sensing the human-understandable phrases and sentences while the random prompts lead to the difficulty in image-text alignment and yield inaccurate semantic representations. However, the form of 'a photo of a [CLS]' performs better than frozen random parameters because this meaningful prompt is more consistent with the input of the original CLIP so that the generated representations can be easily aligned with image representations.
Table 13: Evaluations on frozen parameters of prompts. "Acc." is short for "Accuracy".
ImageNet	Top-1↑	Insertion↑	Deletion↓	CIFAR-10	Full, Acc.	4000, Acc.
Random	75.88	53.3	17.8	Random	94.9	79.5
Fixed	76.20	55.2	17.4	Fixed	95.4	80.7
Learnable (ours)	76.27	57.1	15.1	Learnable (ours)	95.8	81.5