SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 11: Scaling effect on reducing image tokens. We report top-1 zero-shot accuracy on ImageNet-1k [15] classification. N/A: we adopt  $50\%$  and  $75\%$  masking ratio for grid mask, larger masking ratio is non-trivial. To ensure a fair comparison, we keep the length of text tokens constant at 32.
Masking strategy	Image	Text	S/16	B/16	L/16
pre-train	fine-tune	pre-train	fine-tune	pre-train	fine-tune
truncation	112 × 112	32	50.1	54.7	59.0	62.9	65.1	68.9
random	112 × 112	32	50.0	54.8	59.1	62.6	65.3	68.6
block	112 × 112	32	50.4	54.6	59.1	62.9	65.2	68.7
syntax	112 × 112	32	50.1	54.9	58.7	62.6	65.0	68.3
truncation	112 × 112	16	50.6	55.1	58.7	62.4	65.4	68.8
random	112 × 112	16	49.8	54.5	58.4	62.2	65.1	68.5
block	112 × 112	16	50.1	54.5	59.1	63.2	65.3	68.7
syntax	112 × 112	16	50.2	54.7	58.9	63.0	65.3	68.8
truncation	112 × 112	8	45.7	54.2	54.7	62.2	62.2	68.2
random	112 × 112	8	44.5	53.2	54.2	61.5	61.6	67.8
block	112 × 112	8	45.4	54.1	54.0	62.1	61.6	68.2
syntax	112 × 112	8	46.7	54.6	55.6	62.9	62.3	69.0
truncation	112 × 112	6	30.9	52.9	39.0	60.8	47.9	67.1
random	112 × 112	6	299	52.1	38.4	59.7	48.0	66.8
block	112 × 112	6	29.3	52.9	38.3	61.0	46.8	67.8
syntax	112 × 112	6	31.1	53.7	39.7	61.8	49.3	68.4
truncation	112 × 112	4	24.1	49.0	32.6	57.7	40.4	63.6
random	112 × 112	4	22.0	48.9	29.3	57.1	39.6	63.6
block	112 × 112	4	24.0	50.3	31.5	58.8	39.6	65.8
syntax	112 × 112	4	24.7	51.5	32.2	59.6	39.6	66.3
Table 12: Scaling effect on reducing text tokens. We report top-1 zero-shot accuracy on ImageNet-1k [15] classification. Bold represents the best performance among different reducing strategies.
Ablation on fine-tuning schedule and masking. In addition to random masking, we hereby investigate how grid masking and block masking affect fine-tuning performance. The results are reported in Table 9. Interestingly, compared to fine-tuning input tokens at the full resolution, we observe that  $25\%$  masked random fine-tuning and block fine-tuning all lead to a slight performance improvement. With a larger masking ratio, all these masking strategies will lead to worse performance than full-resolution fine-tuning, but overall, random masking consistently yields stronger performance than the other two masking strategies.
Image size	ViT-S/16	ConvNeXt-T	ViT-B/16	ConvNeXt-B
pre-train	fine-tune	pre-train	fine-tune	pre-train	fine-tune	pre-train	fine-tune
224 × 224	-	56.7	-	56.7	-	64.2	-	64.0
160 × 160	54.0	56.0	55.5	56.2	62.2	63.6	63.0	63.6
112 × 112	50.1	54.7	52.6	55.3	59.0	62.9	61.0	63.1
96 × 96	48.3	53.9	51.0	54.6	57.0	62.1	59.2	62.2
64 × 64	40.7	50.5	44.9	51.1	51.0	59.9	54.6	60.0
Table 13: Comparison of ConvNeXt [32] and ViT [18]. We report top-1 zero-shot accuracy on ImageNet-1k [15] classification. To ensure a fair comparison, we keep the length of text tokens constant at 32 and only vary the visual backbones. The associated text encoders are specified in Tab. 4.
We next ablate different fine-tuning setups and summarize the results in Table 10. We choose  $30\%$  masked random fine-tuning as the default strategy, as it leads to a slight performance improvement  $(+0.1\%)$  and enables a  $1.3\times$  speedup of the fine-tuning process. Furthermore, adopting a  $4\times$  fine-tuning schedule results in an additional improvement of  $0.6\%$ . However, we empirically find that further increasing the fine-tuning schedule does not lead to any substantial performance gains.
Following [24], we also investigate progressively fine-tuning with large image resolutions. Initially, for the first 512 million samples, we fine-tune the model using a  $224 \times 224$  input size with a masking ratio of  $30\%$ ; subsequently, for the remaining 128 million samples, we adopt a larger  $336 \times 336$  input size with a masking ratio of  $40\%$  and a smaller learning rate. As shown in the last row of Table 10, i.e., case (5), progressive fine-tuning results in a slight performance improvement of  $0.2\%$  compared to direct fine-tuning with a  $336 \times 336$  input size and meanwhile achieving a notable  $1.5 \times$  speedup of the fine-tuning process.
model	NegCLIP	VG-Relation	VG-Attribute	COCO-Order	Flickr30k-Order
OpenCLIP-B/16	44.7	59.9	41.8	45.3
OpenCLIP-B/16	✓	78.6 (+33.9)	69.5 (+9.6)	87.6 (+45.8)	89.1 (+43.8)
CLIPA-B/16	43.8	57.1	37.8	39.1
CLIPA-B/16	✓	78.5 (+34.7)	68.0 (+10.9)	86.1 (+48.3)	87.9 (+48.8)
Table 14: Comparison on ARO benchmark.
B.3 ImageNet-1k
Image. For reference, Tab. 11 presents the numerical zero-shot ImageNet-1k top-1 accuracy of Fig. 4 with various token length reduction strategies. We can see that fine-tuning plays a crucial role with reduced input token length during pre-training, by comparing the performance of pre-trained and fine-tuned models. For instance, fine-tuning pre-trained models with only 17 tokens (the last four rows in Tab. 11) leads to significant performance gains of  $18.4\%$ ,  $17.0\%$ , and  $18.6\%$  across S/16, B/16, and L/16 scales, respectively, for block masking.
Text. For reference, Tab. 12 presents the numerical zero-shot ImageNet-1k top-1 accuracy of Fig. 7 with various token length reduction strategies. We standardize the visual input size to  $112 \times 112$  pixels for all models and vary only the text input during pre-training. Our fine-tuning procedure follows the same approach as that described for the image input. To ensure a fair comparison, we employ the same masking strategy during fine-tuning as used during pre-training when comparing different masking strategies. Note that the pre-training performance is noticeably lower for input texts with a length smaller than 8. This is because the prompt templates we used for evaluation are often longer than a text length of 8. However, after fine-tuning the model with a maximum length of 32, the models performances are significantly improved.
ConvNeXt [32]. In Tab. 13, we compare the performance of different visual backbones, ViT [18] and ConvNeXt [32]. Notably, we observe that when comparing pre-training results, ConvNeXt outperforms ViT with smaller input sizes, suggesting that CNNs may exhibit greater robustness with respect to scale. For instance, at an input size of  $64 \times 64$ , ConvNeXt-B outperforms ViT-B by approximately  $3.5\%$ . However, after fine-tuning, we observe that the performance gap between the two models narrows considerably across all scales.
B.4 Zero-shot retrieval and robustness
Text. For reference, we also report the performance of zero-shot image/text retrieval on the COCO dataset [30] and zero-shot robustness on the ImageNet-V2 [47], ImageNet-R [22], ImageNet-A [23], and ImageNet-Sketch [58] dataset in Tab. 15 when varying input text token lengths.
masking strategy	image	text	S/16	B/16	L/16
IN-1k	IN-V2	IN-A	IN-R	IN-S	I-to-T	T-to-I	IN-1k	IN-V2	IN-A	IN-R	IN-S	I-to-T	T-to-I	IN-1k	IN-V2	IN-A	IN-R	IN-S	I-to-T	T-to-I
truncation	112 × 112	32	54.7	47.1	14.4	63.0	40.1	31.3	47.7	62.9	54.7	25.2	73.6	48.4	36.6	55.3	68.9	60.5	36.3	80.8	55.5	41.5	59.9
random	112 × 112	32	54.8	46.8	14.1	63.3	40.7	30.7	48.0	62.6	54.8	25.3	74.0	48.8	37.0	54.4	68.6	61.3	36.2	80.2	55.3	41.2	59.0
block	112 × 112	32	54.6	47.3	13.9	62.9	40.6	31.1	48.0	62.9	54.6	25.9	74.0	49.0	36.7	54.2	68.7	60.8	36.7	81.0	55.5	41.5	59.2
syntax	112 × 112	32	54.9	46.6	14.2	63.4	41.1	30.9	49.2	62.6	54.8	25.6	73.5	48.3	37.2	55.3	68.3	60.5	35.3	80.7	55.4	41.3	59.7
truncation	112 × 112	16	55.1	47.2	14.3	63.0	40.8	31.1	47.8	62.4	54.1	25.7	73.5	49.2	36.8	54.9	68.8	61.3	37.1	80.3	55.4	41.3	59.8
random	112 × 112	16	54.5	46.9	14.5	63.2	40.3	30.3	48.8	62.2	54.8	25.3	73.2	48.60	36.6	54.7	68.5	60.6	37.2	81.3	55.9	40.7	59.7
block	112 × 112	16	54.5	46.9	14.5	62.6	40.2	31.1	49.1	63.2	55.3	25.6	73.3	48.6	37.1	55.0	68.7	60.0	37.2	80.5	55.7	41.2	59.4
syntax	112 × 112	16	54.7	47.4	13.4	62.7	40.5	30.8	48.5	63.0	55.4	25.5	73.5	49.0	37.1	55.5	68.8	61.4	37.6	80.7	55.6	41.1	58.6
truncation	112 × 112	8	54.2	46.0	14.4	63.5	40.1	29.7	47.8	62.2	54.9	25.7	74.1	48.3	35.5	53.1	68.2	61.0	37.0	80.5	55.8	39.6	56.7
random	112 × 112	8	53.2	45.6	13.7	62.6	39.1	28.6	47.1	61.5	53.5	24.8	72.6	47.2	34.6	52.6	67.8	59.7	36.2	80.1	55.1	37.8	55.7
block	112 × 112	8	54.1	46.8	13.8	62.8	40.0	30.1	47.9	62.1	54.2	25.0	73.4	47.7	35.5	54.7	68.2	60.8	37.1	80.9	55.8	40.4	57.5
syntax	112 × 112	8	54.6	46.6	13.6	63.5	40.1	29.9	47.7	62.9	54.7	25.7	73.4	49.1	35.4	53.5	69.0	61.6	39.1	81.1	56.3	39.7	57.7
truncation	112 × 112	6	52.9	45.9	13.3	63.6	39.3	28.5	45.9	60.8	53.5	24.2	73.2	47.7	34.4	51.4	67.1	59.5	36.3	80.5	54.8	38.2	54.9
random	112 × 112	6	52.1	44.4	12.1	61.5	38.0	27.4	44.8	59.7	51.9	22.1	71.8	47.0	32.6	50.6	66.8	59.4	35.2	79.7	54.5	36.9	54.3
block	112 × 112	6	52.9	45.6	13.6	62.2	39.0	29.0	45.8	61.0	53.4	24.1	72.7	47.5	34.2	51.6	67.8	60.8	36.3	80.1	55.1	39.2	57.2
syntax	112 × 112	6	53.7	45.8	13.3	63.7	39.8	28.8	46.4	61.8	54.2	25.3	74.0	48.4	34.3	52.4	68.4	61.0	37.6	81.5	56.3	38.5	56.4
truncation	112 × 112	4	49.0	41.6	12.2	61.5	36.3	24.9	40.6	57.7	50.2	21.3	71.8	44.8	30.6	47.2	63.6	56.3	33.6	79.1	52.9	34.2	49.8
random	112 × 112	4	48.9	41.7	11.4	59.2	35.2	25.0	40.6	57.1	49.4	19.0	69.8	44.2	29.6	46.1	63.6	55.2	30.5	77.6	52.4	33.2	49.0
block	112 × 112	4	50.3	42.3	12.5	60.4	37.1	26.0	41.6	58.8	50.8	21.0	71.5	46.0	31.9	49.2	65.8	57.8	33.0	79.7	54.3	35.8	52.9
syntax	112 × 112	4	51.5	44.4	13.4	63.7	38.0	26.2	42.6	59.6	52.5	22.5	72.7	46.8	31.5	48.4	66.3	58.2	35.6	80.7	54.8	35.1	51.7
Table 15: Zero-shot image/text retrieval and robustness. We also study the scaling effect on text tokens on image/text retrieval tasks and robustness benchmarks.