SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xianhang Li* Zeyu Wang* Cihang Xie
* equal contribution
CLIP, one of the pioneering foundation models that connect images and text, has enabled many recent breakthroughs in computer vision. However, its associated training cost is prohibitively high, imposing a significant barrier to its widespread exploration. In this paper, we present a surprising finding that there exists an inverse scaling law for CLIP training, whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. Moreover, we showcase that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law.
As a result of this finding, we are able to successfully train CLIP even with limited computational resources. For example, using 8 A100 GPUs, our CLIP models achieve zero-shot top-1 ImageNet-1k accuracies of  $63.2\%$  in  $\sim 2$  days.  $67.8\%$  in  $\sim 3$  days, and  $69.3\%$  in  $\sim 4$  days. Our method also works well when scaling up — with G/14, we register a new record of  $83.0\%$  ImageNet-1k zero-shot accuracy, and meanwhile accelerate the training by  $\sim 33\times$  compared to its OpenCLIP counterpart. By reducing the computation barrier associated with CLIP, we hope to inspire more research in this field, particularly from academics. Our code is available at https://github.com/UCSC-VLAA/CLIPA.
Foundation models [56, 18, 43, 42] have emerged as a key driving force behind recent breakthroughs in multiple fields, including natural language processing [44, 6, 37, 12], computer vision [68, 46, 25, 50], and robotics [14], and have enabled groundbreaking real-world applications such as ChatGPT [38] and Stable Diffusion [49]. However, the development, training, and deployment of these models present significant challenges due to their high computational resource requirements and the need for specialized technical expertise, consequently restricting accessibility to a small group of researchers and technology companies.
We hereby focus on studying CLIP [42], one of the pioneering foundation models [25, 72, 27, 16, 51, 35] that bridge the gap between text and images and propels computer vision research into the "post-ImageNet" era. The impact of CLIP has been profound, not only in significantly advancing models' zero/few-shot capabilities and out-of-distribution generalization [42], but also in driving the development of the next generation of image-text foundation models, such as DALL-E [46] and Flamingo [2]. Although CLIP training is conceptually simple, reproducing CLIP has been challenging for researchers for years.
To increase the accessibility of CLIP, two significant milestones have been achieved: the OpenCLIP [24] project, which open-sourced the implementation of CLIP, and the release of LAION-400M and LAION-5B datasets [52], providing a wealth of high-quality image-text pairs for training. Yet, despite these strides, the cost of training associated with CLIP remains prohibitively high. For instance, replicating OpenCLIP-B/32's  $62.9\%$  zero-shot top-1 ImageNet-1k accuracy necessitates 36 hours of training with 128 A100 GPUs [24]. This cost is projected to rise considerably with the scaling law
Figure 1: The inverse scaling law for CLIP training. It indicates that larger image/text encoders enable training with fewer image/text tokens while maintaining competitive performance.
[42, 11], which suggests that model performance typically scales proportionally with model size and the number of training tokens, thereby limiting the ability to explore CLIP more broadly and making it challenging for researchers to replicate and build upon these groundbreaking results.
In this paper, we report a surprising finding related to CLIP training that reveals an inverse scaling law. Specifically, we demonstrate that larger image/text encoders allow for the use of shorter image/text token sequences during CLIP training, with only a minor impact on performance. As illustrated in Fig. 1, while a small model S/16 requires a minimum image/text token length of 101/16 to avoid a noticeable performance drop (e.g., within  $1\%$  in zero-shot ImageNet-1k [15] accuracy) compared to the vanilla training with the full token resolution, scaling up to L/16 can significantly reduce this requirement to a minimum image/text token length of 50/6. Additionally, it is worth noting that the strategy for reducing image/text tokens is critical, and those that maximize the retention of original (semantic) information tend to yield better scaling effects.
As a byproduct of this observation, we introduce CLIPA, a framework that can train CLIP efficiently and effectively at scale. For example, by training our CLIPA-L/16 for  $\sim 3$  days on a server with eight A100 GPUs, it achieves a highly competitive  $67.8\%$  zero-shot top-1 accuracy on ImageNet-1k. This performance stands in stark contrast to OpenCLIP-B/16, which attains a  $67.1\%$  zero-shot top-1 accuracy on ImageNet-1k but requires  $\sim 61$  hours of training on 176 A100 GPUs [24], thereby costing over  $16\times$  more GPU hours than our CLIPA-L/16. Our CLIPA can accelerate training more with bigger models — with G/14, CLIPA not only runs  $\sim 33\times$  faster than OpenCLIP in training, but also impressively registers a record-high ImageNet-1k zero-shot top-1 accuracy of  $83.0\%$ .
We hope this research will encourage a more diverse group of researchers, particularly those with limited computation resources, to delve into the exploration of CLIP training, or the training of foundation models in general.
Contrastive Language-Image Pre-training. Over the past few years, the advent of CLIP [42] and ALIGN [25] has transformed the field of visual feature learning through language supervision. By exploiting vast quantities of web-scale data, these pioneering foundation models have shown exceptional zero-shot and out-of-distribution capabilities [42, 65, 70]. The streamlined design of CLIP has facilitated scaling to an unprecedented degree, resulting in substantial performance improvements. As a result, CLIP has been instrumental in empowering a wide range of applications, spanning from segmentation [64], video understanding [62], and image generation [40], to 3D understanding and manipulation [71, 57]. Furthermore, CLIP has played a vital role in catalyzing the development of next-generation image-text foundation models [46, 2, 49].
Efficient CLIP Training. The unparalleled success of CLIP hinges on the scale of both the data [42, 52, 25, 7, 66, 73] and the model [65, 37, 54]. While CLIP adheres impeccably to the scaling law [42, 11], it has also inadvertently sparked a race in large-scale training, one that is seemingly beyond the reach of many researchers in the field. This motivates the development of numerous efficient CLIP training methods. From the data perspective, de-replicating [41, 59, 1], re-sampling [61, 19, 31], and automated data curation [63] have been crucial in creating smaller but high-quality training datasets for accelerating training. On the other hand, FLIP [29] borrows the idea of masking
Figure 2: Visual comparison of different strategies for reducing image token length.
from Masked Image Modeling [21], and removes a large portion of the input image patches (50-75%) for fast CLIP training. The concurrent work RECLIP [28] shows resizing input images into a smaller size is a more effective strategy in speeding up training. Our work is based on FLIP, but it goes a step further by 1) exploring more effective semantic-preserving strategies for token length reduction in CLIP training; 2) pushing the idea of token length reduction to an extreme (i.e., with only 17 images tokens and 8 text tokens), yielding a significant increase in training acceleration (up to  $25 \times$ ).
Scaling law for Language Models. The scaling law has emerged as a powerful tool, linking language model performance with model size, training data, and computational resources with a power-law relation [26]. This conclusion is empirically supported by the GPT model series [6, 37], T-5 [45, 13] and PaLM [12, 3] model families. In this paper, we focus on the scaling behavior of CLIP, but with two critical differences: 1) while the sample efficiency in the language model's scaling law is realized by using few training samples, we probe it by using fewer tokens in each image-text pair in CLIP training; 2) rather than comparing models of different sizes, our observation focuses on performance drop of the same model trained with input of various token lengths.
3 Reducing Image/Text Tokens
We study a total of eight token reduction strategies for CLIP training, four for image-based and four for text-based. Although many of these strategies have been extensively studied in the context of masked image/language modeling, such as random masking, which is generally the most effective, we observe that their effects on CLIP training are distinct.
3.1 Training Setup
Our training setup largely follows FLIP [29]. We use the vanilla ViT [18] as our visual encoder and the non-autoregressive Transformer [56] architecture as our text encoder. We train our models on the LAION-400M [52] dataset for 6.4 epochs, equivalent to  $\sim 2,000$  ImageNet-1k epochs; this is then followed by a 0.36-epoch fine-tuning stage on full-resolution images  $(224\times 224)$  with a maximum text length of 32. To ensure effective contrast between training samples, we set the batch size to  $32k$ . We apply a base learning rate of 8e-6 in the main training stage and 4e-7 in the fine-tuning stage. Gradient Checkpointing [8] is used to conserve GPU/TPU memory. Our data augmentation includes a simple random resizing crop with a minimum cropping ratio of  $40\%$ . Detailed hyperparameter settings and model configurations can be found in the appendix. We train L/16 CLIP models using various token reduction strategies and report the corresponding zero-shot top-1 accuracy on ImageNet-1k [15].
3.2 Image
We start our exploration with FLIP [29], which employs the random masking strategy from MAE [21] to reduce image token length during CLIP training. By setting the masking ratio to  $75\%$ , our re-implementation effectively reports a zero-shot top-1 ImageNet-1k accuracy of  $67.6\%$ .
In addition to random masking, we investigate two other strategies studied in MAE: grid masking, which preserves one patch in each  $2 \times 2$  grid window, and block masking, which removes large blocks from the input. Fig. 2 provides visual examples of these three strategies at a  $75\%$  masking ratio. Intriguingly, while MAE deems random masking as the best strategy for masked image modeling, we notice that CLIP training has a differing preference. For example, grid masking attains a competitive zero-shot top-1 ImageNet-1k accuracy of  $67.3\%$ , while block masking is the most effective, achieving a zero-shot top-1 ImageNet-1k accuracy of  $68.5\%$ .
Figure 3: Visual comparison of different strategies for reducing text token length.
Analysis. We attribute this preference discrepancy to the two tasks' distinct learning natures. In masked image modeling, the objective is to generate absent information from a masked input. Therefore, strategies like random masking that effectively minimize retained information are preferred. In contrast, CLIP training aims to maximize information extraction from the input to achieve better discrimination between different samples. Strategies like block masking, which tend to preserve more structured patterns, can help models yield stronger performance.
Resizing. Building upon this analysis, we propose to apply image resizing as a more direct solution to retaining full image information. We use anti-aliasing bilinear interpolation as the resizing method to best preserve image quality. By training with the image resized to  $112 \times 112$  (which is computationally equivalent to  $75\%$  masking), the L/16 model achieves a zero-shot top-1 ImageNet-1k accuracy of  $68.9\%$ . Notably, this simple resizing strategy surpasses all different mask strategies, highlighting the importance of retaining full input information in CLIP training.
3.3 Text
We next investigate how different strategies for reducing text tokens impact CLIP training. To speed up training, we default to resizing images to  $112 \times 112$  as the image input. We begin with two techniques previously explored in FLIP: truncation and random masking. Truncation selects the first  $N$  text tokens and discards the rest, while random masking randomly drops a portion of the text tokens. An illustrative example of these two strategies with a token length of 4 is shown in Fig. 3. By setting a maximum text token length of 8, truncation performs slightly better than random masking, resulting in a performance of  $68.2\%$  vs.  $67.8\%$ .
Block masking. We conjecture that the performance gain of truncation over random masking may be partially attributed to the use of consecutive text inputs. This leads us to investigate the efficacy of block masking, which randomly preserves consecutive text sequences during training. We limit the number of consecutive text tokens after masking to one for simplicity. With a maximum text token length of 8, this strategy achieves a competitive performance of  $68.2\%$ , outperforming random masking by  $0.4\%$ .
Syntax masking. Another potential approach to improving random masking is to assign different masking priorities to parts of speech. Specifically, we prioritize retaining nouns, followed by adjectives, and then other words. We refer to this strategy as syntax masking. With a maximum text token length of 8, syntax masking achieves the best performance among all strategies, recording a zero-shot top-1 ImageNet-1k accuracy of  $69.0\%$ .
In the next section, we systematically analyze how these four image-based strategies, namely, random masking, grid masking, block masking, and image resizing, and four text-based strategies, namely, truncation, random masking, block masking, and syntax masking, scale with varying token lengths across different model sizes.
4 An Inverse Scaling Law
Training setup. Models of three different scales are used: S/16, B/16, and L/16. Each model includes a visual encoder, namely ViT-S/16 (22M parameters), ViT-B/16 (87M parameters), and ViT-L/16 (304M parameters) [18]. In addition, we use text encoders with 33M, 53M, and 109M parameters, respectively. All these models are trained using the same setup outlined in Sec. 3.1, with one exception that a larger learning rate of 8e-7 is utilized during fine-tuning for S/16 and B/16.
4.1 Image
We first ablate how varying image token lengths affect CLIP training. Specifically, for random masking, block masking, and image resizing, we range the image token length from the full resolution
Figure 4: The inverse scaling law on image tokens. Compared to small models, larger models can utilize fewer image tokens to achieve the same performance drop to the full-resolution baseline.
(196 tokens) to an order of magnitude smaller one (16 tokens); for grid masking, the smallest length is set to 49 (i.e., selecting one in each  $2 \times 2$  window), as it is non-trivial to further reduce it. Note we do not touch the setup for text encoders here, keeping the maximum length for text tokens as 32.
Main Observation. We analyze the zero-shot top-1 accuracy on ImageNet-1k [15] and plot the performance drop compared to the full resolution baseline in Fig. 4. Firstly, we note that performance generally decreases monotonically as token length reduces, which is expected given that models learn less information per sample. The only exceptional case occurs for block masking — when nearly halving the token length from 197 to 99, the performance for L/16 even slightly increases by  $0.3\%$ .
Furthermore, we observe that the performance drop for all four token reduction strategies becomes smaller as the model size increases. For instance, when reducing the token length from 197 to 17 using the resizing strategy, S/16 experiences a  $6.2\%$  performance drop, whereas scaling up the model size to B/16 reduces this drop to  $4.3\%$ ; further using the considerably larger L/16 results in only a  $3.0\%$  performance drop. In other words, it suggests that larger models have the ability to achieve the same performance drop compared to the full-resolution baseline by utilizing fewer image tokens, as compared to their smaller counterparts. We term this phenomenon as the inverse scaling law for CLIP training, implying that by using larger models, we can train with fewer image tokens per sample while still delivering competitive performance.
Lastly, we find that the quality of this inverse scaling law strongly depends on how tokens are removed. More precisely, the more information that is retained, the smaller the length of tokens that can be applied during training. For instance, For instance, with a performance drop threshold of  $2\%$ , random masking requires 99 tokens for B/16 training. However, switching to image resizing, which retains substantially more image information, allows for a significant reduction in the minimum token length, down to 37.
For interested readers, we additionally offer two alternative views to understanding this scaling behavior in Fig. 9 (i.e., model size vs. performance drop) and Fig. 10 (i.e., token number vs. accuracy) in the Appendix.
Zero-shot retrieval. We further evaluate the image/text retrieval performance of CLIP with varying image token lengths on the challenging COCO [30] dataset. Fig. 5 shows the performance drop across different models for four image token reduction strategies. We note that, in most cases, the inverse scaling law proves consistent, as the degree of performance drop gradually decreases with increasing model size. For instance, using the random masking strategy that reduces the token length from 197 to 17, S/16 experiences a performance drop of  $6.6\%$  and  $7.1\%$  for image and text retrieval tasks, respectively. In comparison, the performance drops for B/16 are  $5.8\%$  and  $5.9\%$ , respectively; this performance drop is further reduced to  $4.6\%$  and  $4.1\%$  for L/16.
Zero-shot robustness evaluation. Fig. 6 reports robustness of the aforementioned models, tested on the ImageNet-V2 [47], ImageNet-R [22], ImageNet-A [23], and ImageNet-Sketch [58] datasets. We observe that, in most cases, larger models have a lesser performance drop than small models, which again confirms the validity of this inverse scaling law.
(a) image-to-text
(b) text-to-image
Figure 5: Zero-shot image/text retrieval performance on COCO [30]. Recall@1 is reported.
(a) ImageNet-V2
(c) ImageNet-A
Figure 6: Zero-shot robustness performance.
(d) ImageNet-S
4.2 Text
We next study the impact of altering the maximum text token length on CLIP training. We use all four text reduction strategies introduced in Sec. 3.3, and for each strategy, we range the maximum text token length from 32 to 4. Additionally, to speed up training, we apply a resized  $112 \times 112$  image as input, which runs  $\sim 4 \times$  faster than the  $224 \times 224$  input, while only slightly affecting performance, i.e.,  $0.3\%$  drop on zero-shot top-1 ImageNet-1k accuracy for L/16.
Main observation. The data presented in Fig. 7 reflects a pattern similar to the one observed with image token, i.e., the inverse scaling law is also evident when learning with text tokens. For example, when the maximum text length is set to 4 and the model size is scaled from S/16 to L/16, we observe a decrease in the performance drop from  $5.7\%$  to  $5.2\%$  for truncation,  $3.4\%$  to  $2.0\%$  for syntax masking,  $4.3\%$  to  $2.9\%$  for block masking, and  $5.9\%$  to  $5.1\%$  for random masking. Moreover, our analysis suggests that syntax masking is the most effective strategy for reducing text tokens, especially when setting the maximum text token lengths to be extremely short. For instance, with B/16 and a maximum text token length of 4, all other strategies incur a performance drop of more than  $4.0\%$ , whereas syntax masking results in a performance drop of merely  $3.0\%$ . Furthermore, we observe that for all strategies, the sensitivity of CLIP training to the reduction of text tokens remains relatively low until a threshold of 8 tokens is reached (e.g., the performance drop is less than  $\sim 1.0\%$ ). However, beyond this point, the use of fewer text tokens leads to an abrupt performance drop.
Figure 7: The inverse scaling law on text tokens. Similar to the observation with image tokens, larger models enable training with fewer text tokens while maintaining competitive performance.
Lastly, we notice another intriguing inverse scaling law uniquely related to syntax masking: reducing the text token length from 32 to 16 or 8 consistently enhances the performance of B/16 and L/16 models. This observation suggests that the language signal in our training data may be noisy, and filtering out certain information could potentially facilitate more effective representation learning.
Zero-shot robustness & zero-shot retrieval evaluations. We observe a similar trend for zero-shot robustness evaluation, where larger models typically yield smaller relative performance drops. In terms of zero-shot retrieval performance, for all four text token reduction strategies, we make two interesting observations: 1) there is almost no performance drop for all models when the text token length is reduced to 16; 2) further reducing the text token length to 8 or less, scaling up model size does not evidently help to reduce the performance drop. This second observation is expected, as reducing text length directly affects the capability to align image and text features in a fine-grained manner. Due to space limitations, we include the detailed results of the zero-shot image/text retrieval performance and the zero-shot robustness in Appendix.
4.3 ConvNeXt
In addition to ViT, we validate whether this inverse scaling law is also apparent within the context of CNN architectures. For this analysis, we select ConvNeXt [32], given its outstanding performance on various visual benchmarks. Although different masking strategies are applicable for ConvNeXt, they can only offer a modest training speedup due to the lack of computationally efficient support for sparse convolution [60]. However, image resizing emerges as a viable strategy for expediting CLIP training with ConvNeXt, as it avoids the need of using sparse convolution [55, 20].
We focus on studying ConvNeXt-T and ConvNeXt-B, which are of a similar scale to ViT-S/16 and ViT-B/16, respectively. We utilize the same training setup as for ViT, and incorporate additional augmentations [9, 10]. The full results are listed in Appendix.
Main Observation. We observe that ConvNeXt-B consistently shows a smaller performance drop than ConvNeXt-T when a smaller input size is applied. By setting a performance drop of  $1.5\%$  as the threshold, we find that while ConvNeXt-T necessitates an input image size of  $112 \times 112$ , scaling to ConvNeXt-B enables further reduction of the input size to  $96 \times 96$ . These observations confirm the existence of the inverse scaling law for ConvNeXt in CLIP training.
5 Training CLIP with Limited Resources
Our discussions in Sec. 4 reveal that larger models have the ability to train with fewer tokens while still preserving competitive performance. This ability brings substantial practical benefits, including improved memory footprint and faster training speed. In this section, we showcase how this inverse scaling law can be leveraged to train CLIP models efficiently and effectively, particularly when computational resources are limited.
Figure 8: Accuracy vs. compute trade-off. The x-axis shows overall training cost, and the y-axis shows corresponding ImageNet-1k zero-shot accuracy. The models are trained with different token lengths, resulting in varying compute costs. * indicates the application of additional color jitter and grayscale augmentation, as well as the use of global average pooling instead of the classification token. These modifications are found to be beneficial for stabilizing training with reduced token lengths in large models.