SlowGuess/NeurIPS_2023_OCR huggingface dataset:
model	samples@image resolution	GPU hours	zero-shot classification	zero-shot retrieval
IN-1k	IN-V2	IN-A	IN-R	ObjectNet	IN-Sketch	COCO	Flickr30k
image	text	image	text
OpenAI-B/32, Our Eval	\( 12.8B@224^2 \)	4600	63.3	55.9	31.6	69.3	44.2	42.3	30.4	50.2	58.9	77.6
OpenAI-B/16, Our Eval	\( 12.8B@224^2 \)	10700	68.3	61.9	49.9	77.7	55.3	48.2	33.1	52.4	62.1	81.9
OpenAI-L/14, Our Eval	\( 12.8B@224^2 \)	50800	75.5	69.8	70.8	87.8	68.9	59.6	36.5	56.4	65.3	85.1
OpenCLIP-B/32, Our Eval	\( 12.8B@224^2 \)	4600	62.9	55.1	21.7	73.4	48.9	49.4	35.3	52.6	61.7	79.0
OpenCLIP-B/16, Our Eval	\( 12.8B@224^2 \)	10700	67.1	59.6	33.2	77.9	51.5	52.4	38.3	55.4	65.5	83.3
OpenCLIP-L/14, Our Eval	\( 12.8B@224^2 \)	50800	72.8	65.4	46.5	84.9	59.9	59.6	43.0	59.7	70.3	87.6
CLIPA-B/16 (I50,T16)	\( 2.56B@112^2+128M@224^2 \)	444	63.2	55.6	26.8	73.2	44.3	48.7	35.2	53.1	58.3	75.3
CLIPA-L/16 (I17,T16)	\( 2.56B@64^2+128M@224^2 \)	628	67.8	60.4	38.3	81.2	52.8	56.4	40.1	58.4	64.0	81.5
CLIPA-L16 (I37,T8)	\( 2.56B@96^2+128M@224^2 \)	826	69.3	61.7	43.6	84.0	55.4	58.7	39.8	56.8	67.5	81.9
Table 1: Training CLIPA with limited resources. CLIPA models are first pre-trained with smaller token lengths with 2.56B training samples and subsequently fine-tuned with full token lengths with 128M epochs on LAION-400M. These models are trained on an 8-A100 GPU machine.  $(\mathrm{IX},\mathrm{TY})$  indicates the model is pre-trained with an image token length of  $X$ , and a maximum text token length of  $Y$ . Image resizing and text truncation are used for token length reduction.
We start by recasting the image resizing results of Fig. 4 in the context of computation vs. performance shown in Fig. 8. In addition to the clear performance advantage of larger models over smaller ones, an interesting observation is that this inverse scaling law offers the potential for faster and more powerful CLIP training. For instance, our L/16 model, using a total image token length of 17, outperforms the standard B/16 model setup (i.e., with a total image token length of 197) by  $2.5\%$ , while achieving a  $1.7 \times$  speedup. This process can be further accelerated by training with fewer text tokens, especially when employing a large text encoder (e.g., as in H/14).
Motivated by the above observations, we introduce an effective and efficient CLIP training strategy: training with a larger model but with reduced input token lengths. This approach, dubbed as CLIPA, enables CLIP training even with academic resources. The training setup of CLIPA follows the protocol outlined in Section 3.1, with the addition of color jitter and grayscale image augmentation [9, 10], and the usage of global average pooling in ViT [29, 21]. To reduce the token length in CLIP training, image resizing and text truncation are used by default. More training details can be found in Appendix. All these models are trained using the OpenCLIP codebase [24] in PyTorch [39] on a machine equipped with 8 NVIDIA A100 GPUs.
As demonstrated in Tab. 1, our CLIPA provides both faster training times and improved performance in comparison to OpenCLIP. For example, our CLIPA-B/16 surpasses the vanilla OpenCLIP-B/32 baseline by  $0.3\%$  on zero-shot ImageNet-1k classification, more importantly, requiring  $\sim 10\times$  fewer GPU hours. Similarly, our CLIPA-L/16 outperforms the vanilla OpenCLIP-B/16 baseline by  $0.7\%$ , yet consumes  $17\times$  fewer GPU hours. Notably, our CLIPA-B/16 can be trained on an 8-A100 GPU machine in  $\sim 2$  days, and CLIPA-L/16 in  $\sim 3$  days, highlighting the efficiency and effectiveness of CLIPA in facilitating CLIP training while preserving competitive performance.
model	data source	samples@image resolution	compute(1e12)	zero-shot classification	zero-shot retrieval
IN-Ik	IN-V2	IN-A	IN-R	ObjectNet	IN-Sketch	COCO	Flickr30k
image	text	image	text
FLIP-H/14, Our Eval	LAION-2B	25.68@2242+ 128M@2242	2.4	78.4	71.7	60.3	90.8	69.4	67.5	49.9	67.0	78.3	93.4
OpenCLIP-H/14	LAION-2B	32B@2242	5.7	78.0	70.8	59.2	89.3	69.7	66.6	49.5	66.0	77.8	90.8
OpenCLIP-G/14	LAION-2B	32B@2242+ 6.7B@2242	29.8	80.1	73.6	69.4	92.2	73.0	68.9	51.4	67.3	79.6	92.9
CLIPA-H/14 (I36,T8)	LAION-2B	12.88@842+ 128M@2242	0.4	77.9	71.4	66.2	91.3	71.1	68.4	49.3	66.9	77.2	91.0
12.8B@842+ 512M@2242+ 128M@3362	0.4	79.1	72.3	71.7	92.7	69.9	70.0	50.2	67.5	78.2	92.3
CLIPA-H/14 (I36,T8)	DataComp-1B	12.88@842+ 128M@2242	0.4	81.5	75.0	76.9	94.3	74.1	72.7	49.1	67.0	75.7	90.6
12.8B@842+ 512M@2242+ 128M@3362	0.4	81.8	75.6	82.7	94.4	77.4	72.8	49.2	67.2	76.3	90.3
CLIPA-G/14 (I36,T8)	DataComp-1B	12.8B@842+ 512M@2242	0.8	82.7	76.9	81.7	95.1	77.1	74.3	50.0	67.9	77.7	91.8
12.8B@842+ 512M@2242+ 128M@3362	0.9	83.0	77.3	85.9	95.4	79.7	74.5	50.4	67.8	78.2	92.1
Table 2: Training CLIPA at scale. CLIPA models are first pre-trained with smaller token lengths with 12.8B pre-training samples and subsequently fine-tuned with full token lengths. The Compute cost is measured in the GFLOPs of the model times the number of samples seen during training. (IX, TY) indicates the model is pre-trained with an image token length of  $X$ , and a maximum text token length of  $Y$ .
H/14 model. We hereby include a bigger model, CLIPA-H/14, for experiments. Note that, here we cut the input text token length from 32 to 8, yielding an additional  $\sim 1.3\times$  training speedup. These results are added to Fig. 8. With an input image size of 84 and a text token length of 8, our CLIPA-H/14 achieves a compelling zero-shot top-1 ImageNet-1k accuracy of  $72.8\%$ . This performance is on par with that of OpenCLIP-L/14, while the total computational requirement is reduced by  $\sim 25\times$ .
6 CLIPA at Scale
In this section, we delve deeper into the scaling behavior of CLIPA with larger models (e.g., G/14) and larger datasets (i.e., LAION-2B [52] and DataComp-1B [19]). We default to the setup of 12.8B pre-training samples. We find that extending the fine-tuning schedule at a resolution of  $224 \times 224$  from 128M to 512M training samples, followed by another 128M samples's training at  $336 \times 336$  resolution, demonstrates a clear improvement with the H/14 model ( $79.1\%$  vs.  $77.7\%$ ). Moreover, our updated fine-tuning schedule incorporates a random masking strategy at both resolutions ( $30\%$  for  $224 \times 224$  and  $40\%$  for  $336 \times 336$ ), which reduces the training overheads by a large margin with little-to-no performance drop. More details can be found in the Appendix.
Main results. As shown in Tab. 2, when trained on the same dataset LAION-2B and with the  $224 \times 224$  resolution, our CLIPA-H/14 attains comparable performance with OpenCLIP-H/14 but merely with  $\sim 1 / 15$  training computations. This signifies a remarkable decrease in cost - e.g., given that the training cost for the reported OpenCLIP result amounts to  $\sim 5,600$  GPU-days, CLIPA could save  $\sim 5,230$  GPU-days. Additionally, compared with FLIP-H/14, our CLIPA-H/14 achieves a better  $79.1\%$  ImageNet-1k performance but can be  $6 \times$  faster.
When continuing scaling our model size to G/14, with the same number of seen samples from the DataComp-1B [19] dataset, we successfully establish a new state-of-the-art open-sourced ImageNet-1k zero-shot accuracy of  $83.0\%$ . Notably, this is achieved with  $\sim 33 \times$  less computation compared with previous OpenCLIP-G/14 model. These findings could potentially pave the way for the training of even larger models on even larger datasets, particularly for those with substantial access to GPU/TPU resources.
To further evaluate the performance of our approach, we also evaluate our CLIPA-H/14 model on the VTAB benchmark [69]. The results are included in Tab. 3. On this highly diverse and challenging set of vision tasks, CLIPA still achieves comparable or even superior performances but with significantly less training cost, demonstrating its good generalizability.
7 Limitation
The recent work [67] shows that CLIP models generally are limited at capturing relationships, attributes, and order information. To give a more comprehensive evaluation, we compare our CLIPA model with OpenCLIP on the ARO benchmark [67], a dataset created to evaluate the ability to understand different types of relationships, attributes, and order information. The results are shown in the Appendix (Tab. 14). We can observe that, while OpenCLIP-B/16 slightly outperforms CLIPA-B/16, the absolute performance of both models remains somewhat limited.
model	samples	compute	IN-1k	Caltech101	Cifar10	Cifar100	Clev-all	Clev_dis	Dmlab	Did	Eurosat	Kitti	Flowers	Pets	Peam	Resics45	Smallnorb_A	Smallnorb_E	Svhn
OpenCLIP-H/14	32B	5.7	78.0	84.9	97.5	84.7	26.8	23.6	14.0	67.8	72.6	11.0	80.1	94.5	54.2	69.9	5.4	11.1	56.3
FLIP-H/14, Our Eval	25.6B	2.4	78.4	84.3	98.2	86.9	16.8	24.7	19.7	67.5	64.5	16.5	81.1	95.3	48.5	70.8	5.4	11.0	48.6
CLIPA-H/14 (I36,T8)	12.8B+512M	0.4	77.9	84.8	98.1	87.4	21.2	24.3	15.5	70.2	66.8	16.0	77.9	93.8	56.0	69.7	5.7	11.6	53.4
CLIPA-H/14 (I36,T8)	12.8B+512M+128M	0.4	79.1	84.6	98.2	86.4	17.5	21.6	14.0	71.4	64.0	15.8	79.6	94.5	58.1	70.3	5.8	13.9	59.6
Table 3: Comparison on VTAB benchmarks by zero-shot top-1 accuracy. All models are trained on the LAION-2B dataset. Entries in bold are best results. Compute is measured in GFLOPs (1e12).
To mitigate this relational understanding issue, a composition-aware hard negative mining strategy (NegCLIP) is introduced in [67]. Note that this strategy is extremely lightweight, and can be seamlessly integrated as an additional fine-tuning stage in enhancing CLIP's text understanding ability. Our results in Tab. 14 also corroborate the efficacy of NegCLIP, e.g., both OpenCLIP and CLIPA nearly double their performance on benchmarks like COCO-Order and Flickr30k-Order. Concerning the initial underperformance on ARO benchmarks, we leave it as a future work.
In this paper, we delve deep into CLIP training. Our investigation unveils an intriguing inverse scaling law, suggesting that larger models require fewer input tokens during training. Moreover, among the eight token reduction strategies we studied, we identify that resizing for image input and syntax masking for text input provides the best overall scaling quality. This finding underscores the crucial role of semantic information preservation in efficient CLIP training. Our findings can enable significantly faster CLIP training with better results, especially given limited resources. We hope that our work could encourage a wider range of researchers, particularly those lacking access to substantial computational resources, to engage more in exploring CLIP training.
9 Broader Impact
Large foundation models trained by language supervision have emerged as a pivotal force driving recent advancements in the language and vision domain. Our discovery of the inverse scaling law has democratized access to this technology, enabling the training of proficient CLIP models on a modest budget. This breakthrough has substantial environmental implications, as it significantly curtails tens of thousands of GPU/TPU hours, thereby reducing energy consumption and associated carbon emissions. It is also worth mentioning that our models are trained on publicly available web-scale datasets [53, 52]. Therefore, the derived weights may inadvertently mirror any bias or harmful contents inherent in the training sets. As such, care should be taken when interpreting the outputs of such models and deploy them in the real-world applications.
This work is supported by a gift from Open Philanthropy, TPU Research Cloud (TRC) program, and Google Cloud Research Credits program.
[1] Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic dedduplication. arXiv preprint arXiv:2303.09540, 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[3] Rohan Anil, Andrew M Dai, Orhan First, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
[4] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision, 2022.
[65] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[66] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
[67] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In ICLR, 2022.
[68] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04560, 2021.
[69] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
[70] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
[71] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In CVPR, 2022.
[72] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference.
[73] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023.
model	Embed dim	Vision Transformer	Text Transformer	# params (M)
layers	width	heads	layers	width	heads	vision	text	total
S/16	384	12	384	6	12	384	6	22	33	55
B/16	512	12	768	12	12	512	8	86	53	141
L/16	768	24	1024	16	12	768	12	303	109	414
H/14	1024	32	1280	16	24	1024	16	631	334	967
G/14	1280	48	1664	16	32	1280	20	1844	672	2516
Table 4: CLIP [42] model configurations used in our paper.
Config	Value
optimizer	AdamW [34]
optimizer momentum	(0.9, 0.95)
batch size	32768
base lr	8e-6
minimal lr	0
warm-up steps	1600
schedule	cosine decay [33]
weight decay	0.2
random crop area	(40, 100)
resize method	bi-linear
color jitter [9]	0.32
temperature init	1/0.07 [24, 29]
Table 5: Pre-training hyper-parameters
A.1 Architectures
Our experimental results are based on specific model configurations shown in Tab. 4, following FLIP [29]. Our visual encoder architecture employs three different scales (S/16, B/16, and L/16) with the same patch size, allowing us to investigate the effect of scaling. In our CLIPA models, we employ vanilla ViT [18] with global average pooling as the visual encoder. The sine-cosine positional embeddings are used in ViT [56]. As for text encoder, we adopt the non-autoregressive Transformer [56, 29] and employ a WordPiece tokenizer [17], which includes a "CLS" token for each input text. To ensure uniformity in the input length, we apply zero-padding to those input texts that are shorter than the maximum token length of our model. For the ConvNeXt [32], we employ the same mode configuration as described in [32], and follow the setting in [24].
A.2 Hyper-parameters
Pre-training. Our CLIPA pre-training configuration is outlined in Tab. 5. Notably, due to limited resources, we use a base learning rate of 8e-6 and a smaller  $32k$  batch size. In addition, we apply a color jitter augmentation of strength 0.32 and probability 0.8, and a gray-scale augmentation of probability 0.2 [36, 9, 10].
Fine-tuning. Following pre-training, we conduct a short-period fine-tuning of the models using full-resolution images of size  $224 \times 224$  and texts with a maximal length of 32. The fine-tuning process consists of 4000 steps with an 800-step warm-up period. The base learning rate for fine-tuning is set to 8e-7, while all other parameters remain the same as those used during pre-training. Note that due to limited computation resources, our CLIPA-B/16 and CLIPA-L/16 are fine-tuned with  $8k$  and  $7k$  batch size. The results of different fine-tuning batch size are shown in Tab. 6. It can be observed that a batch size of  $8k$  already achieves competitive performance compared to a batch size of  $32k$ .
Implementation. We implement two codebases based on JAX [5] and Pytorch [39] respectively. Our JAX codebase is built on Big Vision [4] and our pytorch code base mainly followed OpenCLIP [24]. Most of our experiments are conducted with TPU-V3, except that CLIPA-B/16 and CLIPA-L/16 in Tab. 1 are conducted with A100 GPUs.
CLIPA-H/14 and G/14. For comparison with previous state-of-the-art models in the scaling experiments, we follow the approach in FLIP [29], using a 64k batch size for pre-training, and adjusting the warmup steps to 3200 to mitigate the unstable training of larger models. For fine-tuning at a 224 resolution, we apply random masking for image encoder with a  $30\%$  mask ratio to expedite the
Model	32k	16k	8k	4k	2k
CLIPA-L/16 ((I17,T16)	68.1	67.8	67.7	67.1	66.8
Table 6: Ablation on fine-tuning batch size.
Figure 9: Model size vs. Performance drop. Different lines are for different total numbers of input image tokens per sample.
Figure 10: Total number of pre-training tokens vs. Accuracy.
training process and reduce memory costs. And we utilize a 32k batch size with a learning rate of 4e-7 and train with 512M training samples. In fine-tuning at a 336 resolution, the mask ratio is set at  $40\%$  for both models. We further reduce the base learning rate to 1e-7 and train an additional 128M samples with a batch size of 16k. These two models are trained on a 256-core TPU-V3 pod. We incorporate a model-sharding strategy in our G/14 model, which is based on the T5X implementation [48]. Apart from this, we employ a distributed data-parallel strategy.
A.3 Evaluation Setting
Our evaluation protocol is largely based on the original CLIP paper [42], and we employ the benchmarking tool provided by OpenCLIP [24]. To evaluate our model on ImageNet-1k [15], we use 80 prompt templates for zero-shot testing. Following OpenCLIP [24], we resize the shorter side of the input images to 256, and then perform a center crop of size  $224 \times 224$ . When a larger resolution of 336 is adopted, we directly resize the image into  $336 \times 336$  without cropping.
model	GAP	color-jitter [9]	pytorch-impl.	fine-tuning batch size	pre-train	fine-tune
Baseline	32k	58.3	66.3
✓	32k	59.8	67.8 (+1.5)
✓	32k	58.5	67.1 (+0.8)
✓	✓	32k	59.9	68.1 (+1.8)
✓	✓	7k	59.9	67.6 (+1.3)
✓	✓	✓	7k	60.3	67.8 (+1.5)
B More Results
In this section, we present more detailed results, including an alternative view of inverse scaling law, detailed ablation studies of training details and fine-tuning, and numeric results on ImageNet-1k, which we used to plot Fig. 4 in the main text.
B.1 Alternative view of inverse scaling law
For improved representation, we offer two alternative interpretations of Fig. 4 in Fig. 9 and 10. In Fig. 9, we plot the model size on the x-axis, depicting fractions of token reduction as separate lines. In Fig. 10, the total pre-training tokens are used on the x-axis. In both perspectives, it's apparent that larger models demonstrate a significantly smaller performance decrease when using fewer tokens for pre-training.
Table 7: Training details analysis. We use CLIPA-L/16 (I17, T16) model as baseline and report ImageNet-1k [15] zero-shot top-1 performance. GAP: global average pooling in visual encoder. Pytorch-impl.: we re-implement JAX [5] version and reproduce the results with Pytorch [39] on GPUs
model	# image token	# text token	data source	# seen samples	total compute (×1e11)	IN-1k
CLIPA-L/16	36	8	LAION-400M	2.56B + 128M	0.5	69.3
LAION-400M	2.56B + 128M	0.8	72.8
CLIPA-H/14	36	8	LAION-2B	2.56B + 128M	0.8	74.1
LAION-2B	12.8B + 128M	4	77.9
Table 8: Scaling up CLIPA. Specifically, we explore scaling from the aspects of data, model, and schedule. We pretrain the H/14 model with 36 image tokens  $(84 \times 84)$  and 8 text tokens; for finetuning, we use 256  $(224 \times 224)$  image tokens and 32 text tokens.
B.2 Ablation
Ablation on Training details We present a comprehensive analysis of the training details employed to CLIPA. The results are summarized in Tab. 7. First, using global average pooling in ViT, instead of class token as in [24], leads to a substantial improvement of  $\sim 1.5\%$  in ImageNet-1k zero-shot accuracy. Second, we also observe that incorporating stronger augmentation techniques [9] leads to a  $\sim 0.8\%$  improvement. Together, they yield a notable  $1.8\%$  improvement. It is also worth mentioning that to ensure a fair comparison, we switch to the widely-used OpenCLIP codebase [24], which is implemented in PyTorch [39]. Finally, to accommodate the limited GPU memory, we employ a batch size of  $7k$  for fine-tuning the CLIPA-L/16 model. Our experiments demonstrate that this adjustment results in only a marginal decrease in performance.
Ablation on scaling up. We next investigate the scaling behavior of CLIPA. Specifically, our scaling efforts cover three aspects: model, data, and training schedule. The results are reported in Table 8.
First, we can observe that scaling the model size from L/16 to H/14 boosts the performance from  $69.3\%$  to  $72.8\%$ . Furthermore, we note switching the training dataset from LAION-400M [53] to LAION-2B [52] yields another  $1.3\%$  improvement, suggesting the importance of data diversity. Lastly, by increasing the training schedule by a factor of 5, resulting in a total of  $\sim 13\mathrm{B}$  seen samples, we achieve an impressive performance of  $77.9\%$ . We stress that this scaled version of CLIPA-H/14 model readily outperforms its counterpart in FLIP [29] by  $0.3\%$  while requiring only  $1/3$  of the training budget.
These results confirm the efficiency and effectiveness of training CLIPA at scale. Next, we set this CLIPA-H/14 with  $77.9\%$  performance as our baseline for further ablation in the fine-tuning stage.
masking ratio	random	block	grid
0%	77.9	77.9	77.9
25%	78.2	78.0	77.9
50%	77.7	77.6	77.6
75%	76.2	74.3	76.2
Table 9: Comparison of different masking strategies. The results are obtained on on the LAION-2B dataset with H/14 model.
case	masking ratio	resolution	# seen samples	training FLOPs	IN-1k
baseline	0%	2242	128M	177.0G	77.9
(1)	30%	2242	128M	135.9G	78.0
(2)	30%	2242	512M	135.9G	78.6
(3)	30%	2242	640M	135.9G	78.5
(4)	40%	3362	640M	237.8G	78.9
(5)	30%+40%	2242 + 3362	512M+128M	156.3G	79.1
Table 10: Ablation on fine-tuning schedule and masking. In case (5), we use  $224 \times 224$  input with a masking ratio of  $30\%$  for the first 512M samples, and  $336 \times 336$  input with a masking ratio of  $40\%$  for the rest 128M samples.
Masking strategy	Masking ratio	# of tokens	S/16	B/16	L/16
pre-train	fine-tune	pre-train	fine-tune	pre-train	fine-tune
baseline	0.0%	197	-	56.7	-	64.2	-	69.2
random	50.0%	99	54.7	55.5	61.9	62.6	68.3	68.5
grid	50.0%	99	53.9	54.6	62.5	62.8	68.3	68.5
block	50.0%	99	54.9	55.6	63.2	63.5	69.2	69.5
resize	160 × 160	101	54.0	56.0	62.2	63.6	67.8	69.0
random	75.0%	50	49.5	52.7	58.51	60.9	65.9	67.6
grid	75.0%	50	49.5	53.1	57.9	60.7	65.4	67.3
block	75.0%	50	45.2	53.4	57.3	61.4	65.2	68.5
resize	112 × 112	50	50.1	54.7	59.0	62.9	65.1	68.9
random	81.6%	37	47.3	51.6	55.3	58.9	64.1	66.3
grid	81.6%	37	N/A	N/A	N/A	N/A	N/A	N/A
block	81.6%	37	43.6	51.7	54.8	60.7	63.1	67.6
resize	96 × 96	37	48.3	53.9	57.0	62.1	63.8	68.1
random	91.8%	17	36.4	47.5	44.2	55.3	55.3	62.4
grid	91.8%	17	N/A	N/A	N/A	N/A	N/A	N/A
block	91.8%	17	28.4	46.8	38.3	55.3	49.6	62.9
resizing	64 × 64	17	40.7	50.5	51.0	59.9	58.3	66.2