An Inverse Scaling Law for CLIP Training
Authors: Xianhang Li
Zeyu Wang
Cihang Xie
Venue: NeurIPS 2023 poster (2023)
Keywords: CLIP
inverse scaling
efficient training
TL;DR: Unlocking the Power of Inverse Scaling Law: Train a Competitive CLIP Foundation Model (67.8% Zero-Shot ImageNet-1k Accuracy) from Scratch in Just 3 Days on a Single 8-GPU Machine!

Abstract: CLIP, one of the pioneering foundation models that connect images and text, has enabled many recent breakthroughs in computer vision. However, its associated training cost is prohibitively high, imposing a significant barrier to its widespread exploration. In this paper, we present a surprising finding that there exists an inverse scaling law for CLIP training, whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. Moreover, we showcase that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law.


As a result of this finding, we are able to successfully train CLIP even with limited computational resources. For example, using 8 A100 GPUs, our CLIP models achieve zero-shot top-1 ImageNet-1k accuracies of 63.2% in ~2 days, 67.8% in ~3 days, and 69.3% in ~4 days. Our method also works well when scaling up --- with G/14, we register a new record of 83.0% ImageNet-1k zero-shot accuracy, and meanwhile accelerate the training by ~33x compared to its OpenCLIP counterpart.
By reducing the computation barrier associated with CLIP, we hope to inspire more research in this field, particularly from academics. 
Our code is available at https://github.com/UCSC-VLAA/CLIPA.

An Inverse Scaling Law for CLIP Training
Xianhang Li"
Zeyu Wang"
Cihang Xie
"quul contribution UC Santa Cruz
Abstract
CLIP, one of the pioneering foundation models that connect images and text, has enabled many recent breakthroughs in computer vision. However, its associated training cost is prohibitively high, imposing a significant barrier its widespread exploration. In this paper, we present a surprising finding that there exists an inverse scaling law for CLIP training, whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. Moreover, we showcase that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law.
As a result of this finding, are able to successfully train CLIP even with limited computational resources. For example, using 8 A A100 GPUs, our CLIP models achieve zero-shot top-1 ImageNet- Ik accuracies of 63.2% % in ~2 days. 67.8% in x3 days, and 69.3 % in - days. Our method also works well when scaling up with G/14, we register new record of 83.0% ImageNet- Ik zero-shot accuracy, and meanwhile accelerate the training by ~33x compared to its OpenCLIP counterpart. By reducing the computation barrier associated with CLIP, hope inspire more research this field, particularly from academics. Our code is available at https: tutuuuu..a. www.com////oo// wwm/UCCC- VLAA/CLIPA.
Introduction
Foundation models [56, 18, 43, 42] have emerged key driving force behind recent breakthroughs in multiple fields, including natural language processing [44, 6, 37, 12], computer vision [68, 46, 25, 50], and robotics [14], and have enabled groundbreaking real-world applications such as ChatGPT [38] and Stable Diffusion [49]. However, the development, training, and deployment of these models present significant challenges due to their high computational resource requirements and the need for specialized technical expertise, consequently restricting accessibility to a small group of researchers and technology companies.
We hereby focus on studying CLIP [42], one the pioneering foundation models [25, 72, 27, 16, 51, 35] that bridge the gap between text and images and propels computer vision research into the ppost-ImageNet" era. The impact of CLIP has been profound, not only in significantly advancing models' zero/few-shot capabilities and of-distributtin generalization [42]. but also in driving the development of the next generation of image-text foundation models, such as DALL-E [46] and Flamingo [2]. Although CLIP training is conceptually simple, reproducing CL.IP has been challenging for researchers for years.
To increase the accessibility of CLIP, two significant milestones have been achieved: the OpenCLIP [24] project, which open-sourced the implementation of CLIP, and the release of LAION-400M and LAION-SB datasets [52], providing wealth of high-quality image-text pairs for training. Yet, despite these strides, the cost of training associated with CLIP remains prohibitively high. For instance, replicating OpenCLIP-BB/B
62.9% zero-shot top-1 1 ImageNet- k accuracy necessitates 36 hours of training with 128 1000 GPUs [24]. This cost is projected to rise considerably with the scaling law
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

contrast
contrast
Image Encoder
Image Encoder
Toxt Encoder
Image Encoder
Text Encodar
Model Size
Encoder
Token Size
An orange cat that is sittngg ledge
sitting
orange
Figure 1: The inverse scaling law for CLIP training. It indicates that larger image/text encoders enable training with fewer image/text tokens while maintaining competitive performance.
[42, 11], which suggests that model performance typically scales proportionally with model size and the number of training tokens, thereby limiting the ability to explore CLIP more broadly and making it challenging for researchers to replicate and build upon these groundbreaking results.
In this paper, we report a surprising finding related to CLIP training that reveals an inverse scaling law. Specifically, we demonstrate that larger image/text encoders allow for the of shorter image/text token sequences during CLIP training, with only a minor impact on performance. As illustrated in Fig. 1, while small model S/16 requires minimum image/text token length of 101/16 to avoid noticeable performance drop (e.g., within 1% in zero-shot ImageNet- 1k [15] accuracy) compared to the vanilla training with the full token resolution, scaling up to L/16 significantly reduce this requirement to a minimum image/text token length 50/6. Additionally, strategy for reducing image/text tokens is critical, and those that maximize the retention of original (semantic) information tend to yield better scaling effects.
worth noting that the
As byproduct of this observation, we introduce CLIPA, framework that can train CLIP efficiently and effectively at scale. For example, by training our CLIPA-L/I6 for ~3 days on a server with eight A100 GPUs, it achieves highly competitive 67.8% zero-shot top-1 accuracy on ImageNet-II Ik. This performance stands in stark contrast to OpenCLIP-B/ which attains 67.1% zero-shot top-1 accuracy on ImageNet-11 Ik but requires ~61 hours of training 176 A100 GPUs [24]. thereby costing over 16x more GPU hours than our CLIPA-L/16 Our CLIPA can accelerate training more with bigger models with G/14, CLIPA not only runs ~33 faster than OpenCLIP in training, but also impressively registers a reih ImageNet-Il Ikk zero-shot top-1 accuracy of 83.0%.
We hope this research will encourage a more diverse group of researchers, particularly those with limited computation resources, to delve into the exploration of CLIP training, or the training of foundation models in general.
2 Related Works
Contrastive Language-Iman Pre-training. Over the past few years, the advent of CLIP [42] and ALIGN [25] has transformed the field of visual feature learning through language supervision. By exploiting vast quantities of web-scale data, these pioneering foundation models have shown exceptional zero-shot and out-of-distribution capabilities [42, 65, 70]. The streamlined design of CLIP has facilitated scaling an unprecedented degree, resulting substantial performance improvements. As result, CLIP has been instrumental in empowering wide range of applications, spanning from segmentation [64], video understanding [62], and image generation [40], 3D understanding and manipulation [71, 57]. Furthermore, CLIP has played vital role in catalyzing the development of next-generation image-text foundation models [46, 49].
Efficient CLIP Training. The unparalleled success of CLIP hinges on the scale of both the data [42, 52, 25, 66, 73] and the model [65, 37, 54]. While CLIP adheres impeccably to the scaling law [42, 11], it has also inadvertently sparked a race in large-scale training, one that is seemingly beyond the reach of many researchers in the field. This motivates the development of numerous efficient CLIP training methods. From the data perspective, de-replicating [41, 59, 1], re-sampling (61, 19, 31], and automated data curation [63] have been erucial creating smaller but high-quality
training datasets for accelerating training. On the other hand, FLIP [29] borrows the idea of masking

---

Original image
Random masking
Grid masking
(d) Block masking
(e) Reaizing
Figure 2: Visual comparison of different strategies for reducing image token length.
from Masked Image Modeling [21], and removes a large portion of the input image patches (50-75%) for fast CLIP training. The concurrent work RECLIP [28] shows resizing input images into smaller size is a more effective strategy in speeding up training. Our work is based on FLIP, but it goes a step further by 1) exploring more effective semmanti--reservving strategies for token length reduction in CLIP training; 2) pushing the idea of token length reduction to an extreme (i.e., with only 17 images tokens and 8 text tokens), yielding significant increase in training acceleration (up to 25x).
Scaling law for Language Models. The scaling law has emerged as powerful tool, linking language model performance with model size, training data, and computational resources with a pawer-law relation [26]. This conclusion is empirically supported by the GPT model series [6, 37]. T-5 [45, 13] and PaLM [12, 3] model families. this paper, we focus on the scaling behavior of CLIP, but with two critical differences: 1) while the sample efficiency in the language model's 's scaling law is realized by using few training samples, we probe by using fewer tokens in each image-text pair in CLIP training; 2) rather than comparing models of different sizes, our observation focuses on performance drop of the same model trained with input of various token lengths.
3 Reducing Image/Text Tokens
We study a total of eight token reduction strategies for CLIP training, four for image-based and four for text-based. Although many of these strategies have been extensively studied in the context of masked image/language modeling, such as random masking, which is generally the most effective, we observe that their effects on CLIP training are distinct.
3.1 Training Setup
Our training setup largely follows FLIP [29]. We use the vanilla VIT [18] as our visual encoder and the non-autoregressivve Transformer [56] architecture our text encoder. We train models on the LAION-400M [52] dataset for 6.4 epochs, equivalent to ~2,000 ImageNet- Ik epochs; this is then followed by 0.36-epoch fine-tuning stage on full-resolution images (224 x 224) with maximum text length of 32. To ensure effective contrast between training samples, we set the batch size to 32k. We apply base learning rate of 8e-6 in the main training stage and 4e-7 in the fine-tuning stage. Gradient Checkpointing [8] is used to conserve GPU/TPU memory. Our data augmentation includes a simple random resizing crop with a minimum cropping ratio of 40%. Detailed hyperparameter settings and model configurations can be found in the appendix. We train L/16 CLIP models using various token reduction strategies and report the corresponding zero-shot top-1 accuracy on ImageNet-Ik [15].
3.2 Image
We start our exploration with FLIP [29], which employs the random masking strategy from MAE [21] to reduce image token length during CLIP training. By setting the masking ratio to 75%, our re-implementation effectively reports a zero-shot top-1 ImageNet-Ik accuracy of 67.6%.
In addition to random masking, we investigate two other strategies studied in MAE: grid masking. which preserves one patch in each 2 grid window, and block masking, which removes large blocks from the input. Fig. 2 provides visual examples of these three strategies at a 75% masking ratio. Intriguingly, while MAE deems random masking as the best strategy for masked image modeling, we notice that CLIP training has a differing preference. For example, grid masking attains a competitive zero-shot top-1 ImageNet- Ik accuracy of 67.3%, while block masking the most effective, achieving zero-shot top-1 ImageNet-Il accuracy of 68.5%.
3

---

An orange cat that is sitting ledge
An orange cat that
orange is on ledge
cat that is sitting
An cat a ledge
(a) original text
(b) truncation
(c) random masking (d) block masking
(c) syutax masking
Figure 3: Visual comparison of different strategies for reducing text token length.
Analysis. We attribute this preference discrepancy to the two tasks' distinct learning natures. In masked image modeling, the objective is to generate absent information from masked input. Therefore, strategies like random masking that effectively minimize retained information are preferred. In contrast, CLIP training aims to maximize information extraction from the input to achieve better discrimination between different samples. Strategies like block masking, which tend to preserve more structured patterns, can help models yield stronger performance.
Resizing. Building upon this analysis, we propose to apply image resizing as a more direct solution to retaining full image information. We use anti-aliasing bilinear interpolation as the resizing method to best preserve image quality. By training with the image resized to 112 x x11 12 (which is computationally equivalent to 75% masking), the L/16 model achieves zero-shot top-1 ImageNet- Ik accuracy of 68.9%. Notably, this simple resizing strategy surpasses all different mask strategics, highlighting the importance of retaining full input information in CLIP training.
3.3 Text
We next investigate how different strategies for reducing text tokens impact CLIP training. To speed up training, we default to resizing images to 112 x 112 as the image input. We begin with two techniques previously explored in FLIP: truncation and random masking. Truncation selects the first N text tokens and discards the rest, while random masking randomly drops a portion of the text tokens. An illustrative example of these two strategies with a token length of 4 shown in Fig. By setting maximum text token length of 8, truncation performs slightly better than random masking, resulting in performance of 68.2% vs. 67.8%.
Block masking, We conjecture that the performance gain of truncation over random masking may be partially attributed the use of consecutive text inputs. This leads us investigate the efficacy of block masking, which randomly preserves consecutive text sequences during training. We limit the number of consecutive text tokens after masking to one for simplicity. With maximum text token length of 8, this strategy achieves competitive performance of 68.2 68.2%, outperforming random masking by 0.4%.
Syntax masking. Another potential approach to improving random masking is to assign different masking priorities to parts of speech. Specifically, we prioritize retaining nouns, followed by adjectives, and then other words. We refer to this strategy syntax masking. With maximum text token length of 8, syntax masking achieves the best performance among all strategies, recording a zero-shot top-1 ImageNet-11 Ik accuracy of 69.0%.
In the next section, we systematically analyze how these four image-based strategies, namely, random masking, grid masking. block masking, and image resizing, and four text-based strategies, namely, truncation, random masking, block masking, and syntax masking, scale with varying token lengths across different model sizes.
4 An Inverse Scaling Law
Training setup. Models of three different scales used: S/16, B/16, and I/16. Each model includes a visual encoder, namely ViT-S/16 (22M parameters), ViT-B/16 (87M parameters), and ViT-L/16 (304M parameters) [18]. In addition, we use text encoders with 33M, 53M, and 109M parameters, respectively. All these models are trained using the same setup outlined in Sec. 3.1, with one exception that larger learning rate of 8c-7 is utilized during fine-tuning for S/16 and B/16.
4.1 Image
We first ablate how varying image token lengths affect CLIP training. Specifically, for random masking, block masking, and image resizing, range the image token length from the full resolution

---

Randon mask
Resize
Block mask
Grid mask
Performanc drop (5)
-2
-2
-2
-2
-6-
-6-
-6
S/1G
5/16
S/16
R/16
R/16
R/16
R/16
10
10
10
10
187
+ 9
Number tokens
Number of tokens
Number of tokens
Number tokens
Figure 4: The inverse scaling law on image tokens. Compared to small models, larger models can
utilize fewer image tokens to achieve the same performance drop to the full-resolution baseline.
(196 tokens) an order of magnitude smaller one (16 tokens); for grid masking, the smallest length is set to 49 (i.c., selecting one in each 2 x 2 window), as it is non-trivial to further reduce it. Note we do not touch the setup for text encoders here, keeping the maximum length for text tokens as 32.
Main Observation. We analyze the zero-shot top-1 accuracy on ImageNet- Ik [15] and plot the performance drop compared to the full resolution baseline in Fig. 4. Firstly, we note that performance generally decreases monotonically as token length reduces, which is expected given that models learn less information per sample. The only exceptional case occurs for block masking when nearly halving the token length from 197 to 99, the performance for L/16 even slightly increases by 0.3%.
Furthermore, we observe that the performance drop for all four token reduction strategies becomes smaller as the model size increases. For instance, when reducing the token length from 197 to 17 using the resizing strategy, S/16 experiences a 6.2% performance drop, whereas scaling up the model size to B/16 reduces this drop to 4.3%; further using the considerably larger 1/16 results in only a 3.0% performance drop. In other words, it suggests that larger models have the ability to achieve the same performance drop compared to the full-resolution bascline by utilizing fewer image tokens, as compared to their smaller counterparts. We term this phenomenon as the inverse scaling law for CLIP training, implying that by using larger models, we can train with fewer image tokens per sample while still delivering competitive performance.
Lastly, we find that the quality of this inverse scaling law strongly depends on how tokens are removed. More precisely, the more information that is retained, the smaller the length of tokens that can be applied during training. For instance, For instance, with a performance drop threshold of 2%, random masking requires 99 tokens for B/16 training. However, switching to image resizing, which retains substantially more image information, allows for significant reduction in the minimum token length, down to 37.
For interested readers, we additionally offer two alternative views to understanding this scaling behavior in Fig. 9 (i.e., model size vs. performance drop) and Fig. 10 (i.e., token number vs. accuracy) in the Appendix.
Zero-shot retrieval. We further evaluate the image/text retrieval performance of CLIP with varying image token lengths the challenging COCO [30] dataset. Fig. 5 shows the performance drop across different models for four image token reduction strategies. We note that, in most cases, the inverse scaling law proves consistent, the degree of performance drop gradually decreases with increasing model size. For instance, using the random masking strategy that reduces the token length from 197 17, S/16 experiences performance drop of 6.6% and 7.1% for image and text retrieval tasks, respectively. In comparison, the performance drops for B/16 are 5.8% and 5.9%. respectively; this performance drop is further reduced 4.6% and 4. 1% for I/16.
Zero-shot robustness evaluation. Fig. reports robustness of the aforementioned models, tested on the ImageNet-V22 V2 [47], ImageNet-R [22]. ImageNet-A [23], and ImageNet-Skette [58] datasets. We observe that, in most cases, larger models have a lesser performance drop than small models, which again confirms the validity of this inverse scaling law.
5

---

Rundom
Resize
Rundon
Block
Peformane (50)
316
SI15
SIE5
SI55
5X76 1016
1016
(a) image to dext
(b) text to image
Figure 5: shot image/text retrieval performance on COCO [30]. Recall @ is reported.
Block mark
Grid mask
Random mask
Performance drop %%
526
2115
5155
525
2016
S155
5-2
Vindoral
Memder
(a) ImageNet-V2
(b) ImageNot-R
2016
2016
210
5/15
2016
316
2116
B016
(c) ImageNet-A
(d) ImageNet-S
Figure 6: Zero-shot robustness performance.
4.2 Text
We next study the impact of altering the maximum text token length on CLIP training. use all four text reduction strategies introduced in Sec. 3.3, and for each strategy, range the maximum text token length from 32 to 4. Additionally, to speed up training, we apply a resized 112 x 112 image as input, which ie., 0.3% drop on zero-shot top-1 ImageNet- 1k accuracy for L/16.
~4x faster than the 224 224 input, while only slightly affecting performance,
Main observation. The data presented in Fig. 7 reflects pattern similar the observed with image token, i.e., the inverse scaling law also evident when learning with text tokens. For example, when the maximum text length a decrease in the performance drop from 5.7% to 5.2% for truncation, 3,4% to 2.0% for syntax masking, 4.3% to 2.9% for block masking, and 5.9% 5.1% % for random masking. Moreover, our analysis suggests that syntax masking is the most effective strategy for reducing text tokens, especially when setting the maximum text token lengths to extremely short. For instance, with B/16 and maximum text token length of 4, all other strategies incur performance drop of more than 4.0%, whereas syntax masking results in a performance drop merely 3.0%. Furthermore, we observe that for all strategies, the sensitivity of CLIP training to the reduction of text tokens remains relatively low until a threshold of 8 tokens reached (e.g., g., the performance drop is than ~1.0%). However, beyond this point, the use of fewer text tokens leads to abrupt performance drop.
to 4 and the model size is scaled from S/16 L/16, we observe

---

Truncation
Random mask
Block musk
Syntax mask
Performance drop (%e
5/16
5/16
L/16
L/16
L/16
L/18
6
9
+0
%
Number tokens
Number of tokens
Number of tokens
Number of tokens
Figure 7: The inverse scaling law on text tokens. Similar to the observation with image tokens, larger models enable training with fewer text tokens while maintaining competitive performance.
Lastly, we notice another intriguing inverse scaling law uniquely related to syntax masking: reducing the text token length from 32 to 16 or 8 consistently enhances the performance of B/16 and L/16 models. This observation suggests that the language signal in our training data may be noisy, and filtering out certain information could potentially facilitate more effective representation learning.
Zero-shot robustness & zero-shot retrieval evaluations. We observe a similar trend for zero-shot robustness evaluation, where larger models typically yield smaller relative performance drops. In terms of zero-shot retrieval performance, for all four text token reduction strategies, we make two interesting observations: 1) there is almost no performance drop for all models when the text token length reduced to 16; 2) further reducing the text token length to or less, scaling up model size does not evidently help reduce the performance drop. This second observation is expected, as reducing text length directly affects the capability to align image and text features a fine- lin-ggained manner. Due to space limitations, we include the detailed results of the zero-shot image/text retrieval performance and the zero-shot robustness in Appendix.
4.3 ConvNeXt
In addition to ViT, we validate whether this inverse scaling law is also apparent within the context of CNN architectures. For this analysis, we select ConvNeXt [32], given outstanding performance on various visual benchmarks. Although different masking strategies they can only offer a modest training speedup due to the lack of computationally efficient support for sparse convolution [60]. However, image resizing emerges training with ConvNeXt, as avoids the need using sparse convolution [55, 20].
applicable for Nt,,
viable strategy for expediting CLIP
We focus on studying ConvNeXt-T and ConvNeXt-B, which are of a similar scale to ViT-S/16 and ViT-B/16, respectively. We utilize the same training setup as for ViT, and incorporate additional augmentations [9, 10]. The full results are listed in Appendix.
Main Observation. We observe that ConyNeXt-B consistently shows smaller performance drop than ConvNeXt-T when smaller input size is applied. By setting performance drop of 1.5% the threshold, we find that while ConvNeXt-T necessitates an input image size of 112 x 112, scaling to ConvNeXt-B enables further reduction of the input size to 96 x 96. These observations confirm the existence the inverse scaling law for ConvNeXt in CLIP training.
Training CLIP with Limited Resources
Our discussions in Sec. 4 reveal that larger models have the ability to train with fewer tokens while still preserving competitive performance. This ability brings substantial practical benefits, including improved memory footprint and faster training speed. In this section, we showcase how this inverse scaling law can be leveraged to train CLIP models efficiently and effectively, particularly when computational resources are limited.
7

---

Zero shot Top-1 ImageNo Accuracy (%)
72.8
Speodup Accuracy: 3.6%
Speedup Aceuray:
1.7x Speedup Accurncy: +.55%
8:
1.4x speedup Accuracy: 3.2%%
3.2%
8/18
8/16
1/16
H114"
Compute GFLOPs Training Samples (1e"))
Figure 8: Accuracy vs. compute trade-ofT. The x-axis shows overall training cost, and the y-axis shows corresponding ImageNet-Ik zero-shot accuracy. The models trained with different token
lengths, resulting in varying compute costs, iudicates the application of additional color jitter and gray scale augmertiation, well the of global average pooling instead the clatsification token. These modifications found to beneficiol for stabilizing rainiin with reduced then lengths in large models.
zero-shot classification
zcro-shot retrieval
COC0
Plickr30k
IN-ik
IN-V2
IN
ObjectNet
IN Skeich
model
samples Gmnage resolution
GPU hours
image
96x1
image
tEst
OpenAl B/32, Our Eval
22.80-62224
4600
63.3
55.9
31.6
69.3
42.3
30.4
50.2
58,9
77.6
OpenA1-B/16, Feal
12.
10700
53.1
OpmnA1-L/14,
12.58.02224
36.5
UpenCLIP-B/32 Our
21.7
35.3
52.6
OpenCIP--/166 Our Eval
(2
10700
67.1
59.6
33.2
77.9
51.5
52.4
38.3
55.4
65.5
83.3
OpeCCLI--L/1, Our Boal
12.8802224
50800
72.8
65.4
46.5
84.9
59.9
59.6
43.0)
99.7
71.3
87.6
CLIPA Drió (150.T16)
2,560 112++128M6824
644
63.2
55.6
26.8
73.2
44.3
48.7
35.2
53.1
58.3
75.3
CLIPA L/16 (177/16))
2.560 3600 224
628
67.8
60.4
38.3
81.2
52.8
56.4
40.1
58.4
64.0
81.5
11
2.568:90--133M0-2
826
69.3
61.7
43.6
84.0
55.4
58.7
59.8
56.8
67.5
81.9
Table 1:
Training CLIPA with limited resources. CLIPA models are first pre-trained with smaller token lengths with 2.56B training samples and subsequently fine-tuned with full token lengths with 128M epochs LAION-40MM. These models are trained on an 8-A100 GPU machine. '(IXTTY) indicates the model is pre-trained with an image token length of X, and a maximum text token length of Y. Image resizing and text truncation are used for token length reduction.
We start by recasting the image resizing results of Fig, in the context of compulation vs. performance shown in Fig. 8. In addition to the clear performance advantage of larger models over smaller ones, an interesting observation is that this inverse scaling law offers the potential for faster and more powerful CLIP training. For instance, our L/16 model, using total image token length of 17, outperforms the standard B/16 model setup (i.e., with a total image token length 197) by 2.5%, while achieving a 1.7x speedup. This process can be further accelerated by training with fewer text tokens, especially when employing a large text encoder (e.g., as in H/14).
Motivated by the above observations, we introduce an effective and efficient CLIP training strategy: training with larger model but with reduced input token lengths. This approach. dubbed as CLIPA, enables CLIP training even with academic resources. The training setup of CLIPA follows the protocol outlined Section 3. with the addition of color jitter and grayscale image augmentation [9, 10], and the usage of global average pooling in VIT [29, 21J. To reduce the token length in CLIP training, image resizing and text truncation are used by default. More training details can be found in Appendix. All these models are trained using the OpenCLIP codebase [24] in PyTorch [39] machine equipped with 8 NVIDIA A100 GPUs
As demonstrated in Tab. 1, our CLIPA provides both faster training times and improved performance in comparison to OpenCLIP. For example, our CLIPA-B/16 surpasses the vanilla OpenCLIP-B/33 baseline by 0.3% on zero-shot ImageNet- k classification, more importantly, requiring ~ ~10x fewer GPU hours. Similarly, our CLIPA-L/16 outperforms the vanilla OpenCLIP-B//B baseline by 0.7%, yet consumes 17x fewer GPU hours. Notably, our CLIPA-B/16 can be trained on an 8-A100 GPU machine in ~2 days, and CII in ~3 days, highlighting the efficiency and effectiveness of CLIPA in facilitating CLIP training while preserving competitive performance.
8

---

zero-dust dasiiiinia
aurt-sher retrioval
wntelo/Ringg remiadionn
compoddel22
18-12
1N-22
Ds-A
18-1
Otyethia
IN-Skridh
Iseage
COCO
texi
inage
Ioll
RI1,, Our Bull
LAION-3B 38
25.88-3502
244
76.4
71.7
603
90.6
09.4
61.5
49.9
67.0
7%,3
95.4
LAIUN -15
1254224
5.7
78.2
70.8
592
80.3
09.7
16.5
40.5
77.8
Opar
LAINN-21
298
85.1
694
91.2
7300
16.0
51.4
TMA
92.9
CLIPA (P677)
LAION
12,38-544
1ERM69224
91.
71.1
DanCCrnp-1
1758107244
12,38:54
512M92226
128M
912
724
76.3
90.3
DesCoop--B IB
512MP224
91.7
95.1
73.
77.7
91.8
12.90 312M66226
128M663362
859
98.4
797
18.2
92.1
Table 2: Training CLIPA at scale. CLIPA models are first pre-trained with smaller token lengths
with 12.88 pre-training samples and subsequently fine-tuned with full token lengths. The Compute cost is measured in the GFLOPs of the model times the number of samples seen during training. "(IX,TY)' indicates the model is pre-trained with an image token length of X, and a maximum text token length of Y.
H/14 model. We hereby include bigger model, CLIPA-H/14, for experiments. Note that, here we cut the input text token length from 32 to 8, yielding an additional ~1.3x training speedup. These results are added Fig. 8. With an input image size of 84 and a text token length of our CLIPA- H/14 achieves a compelling zero-shot top-1 ImageNet- Ik accuracy of 72.8%. This performance is on par with that of OpenCLIP-L/14, while the total computational requirement reduced by 25
6 CLIPA at Scale
In this section, delve deeper into the scaling behavior of CLIPA with larger models (e-g., G/14)
and larger datasets (i.e., LAION-2B [52] and DataComp- 1B [19]). We default to the setup of 12.8B
pre-training samples. We find that extending the fine-tuning schedule at a resolution of 224 x 224 from 128M to 512M training samples, followed by another 128M samples's s training at 336 x 336 resolution, demonstrates a clear improvement with the H/14 model (79.1% vs. 77.7%). Moreover, our updated fine-tuning schedule incorporates random masking strategy at both resolutions (30% for 224 x 224 and 40% for 336 336), which reduces the training overheads by large margin with to- performance drop. More details can be found in the Appendix.
Main results. As shown in Tab. 2, when trained the same dataset LAION-2B and with the 224 x 224 resolution, our CLIPA-H/I4 attains comparable performance with OpenCLIP-H/II but merely with 1/15 training computations. This signifies remarkable decrease in cost e-g., given that the training cost for the reported OpenCLIP result amounts to ~5,600 GPU-days, CLIPA could save ~5.230 GPU-days. Additionally, compared with FLIP-H/14, our CLIPA-H/14 achieves better 79.1% ImageNet- performance but can be 6x faster.
When continuing scaling our model size to G/14, with the same number of seen samples from the DataComp- IB [19] dataset, we successfully establish a new state-of-the-art open-soureed ImageNet- Ik zero-shot accuracy of 83.0% Notably, this is achieved with <33 x less computation compared with previous OpenCLIP- -GI4 model. These findings could potentially pave the way for the training of even larger models on even larger datasets, particularly for those with substantial access to GPU/TPU resources.
To further evaluate the performance of our approach, we also evaluate our CLIPA-H/14 model on the VTAB benchmark [69]. The results are included in Tab. 3. On this highly diverse and challenging set of vision tasks, CLIPA still achieves comparable or even superior performances but with significantly less training cost, demonstrating its good generalizability.
7 Limitation
The recent work [67] shows that CLIP models generally are limited at capturing relationships. attributes, and order information. To give more comprehensive evaluation, we compare our CLIPA model with OpenCLIP on the ARO benchmark [67], a dataset created to evaluate the ability to understand different types of relationships, attributes, and order information. The results are shown in the Appendix (Tab. 14). We can observe that, while OpenCIIP-BB16 slightly outperforms CLIPA- B/16, the absolute performance of both models remains somewhat limited.

---

Calaad
ClG 10
CIG 100
Dhildb
Smallooh
Smallorh
DM
model
Samples
2216
5.7
75.0
$4.9
972
2633
23.5
150
110
953
36.3
ELIP HI14, Our Boal
25
78.4
54.3
98.2
36.9
16.5
24.7
19.7
675
64.5
16.5
8111
95.3
48.5
784
1100
48.6
CLIPA-HI-
CLIPA-II (B6T8))
98.2
14.0
714
Table Comparison on VTAB benchmarks by zero-shot top-1 accuracy. models are trained on the LAION-2B dataset. Entries in bold are best results. Compute is measured in GFLOPs (1e12).
To mitigate this relational understanding issue, composition-awann hard negative mining strategy (NegCIIP) is introduced in [67]. Note that this strategy is extremely lightweight, and can be seamlessly integrated ability. Our results in Tab. 14 also corroborate the efficacy of NegCLIP, e.g., both OpenCLIP and CLIPA nearly double their performance on benchmarks like COCO-Order and Flick33Ok-Order Concerning the initial underperformance on ARO benchmarks, we leave it as a future work.
additional fine-tuning stage enhancing CLIPP's text understanding
8 Conclusion
In this paper, we delve deep into CLIP training. Our investigation unveils intriguing inverse scaling law, suggesting that larger models require fewer input tokens during training. Moreover, among the eight token reduction strategies studied, we identify that resizing for image input and syntax masking for text input provides the best overall scaling quality, This finding underscores the crucial role of semantic information preservation in efficient CLIP training. Our findings can enable significantly faster CLIP training with better results, especially given limited resources. We hope that our work could encourage wider range of researchers, particularly those lacking access to substantial computational resources, to engage more in exploring CLIP training.
Broader Impact
Large foundation models trained by language supervision have emerged as pivotal force driving recent advancements in the language and vision domain. Our discovery the inverse scaling law has democratized access to this technology, enabling the training of proficient CLIP models modest budget. This breakthrough has substantial environmental implications, as it significantly curtails tens of thousands of GPU/TPU hours, thereby reducing energy consumption and associated carbon emissions. It is also worth mentioning that our models are trained on publicly available web-scale datasets [53, 52]. Therefore, the derived weights may inadvertently mirror any bias or harmful contents inherent in the training sets. As such, care should be taken when interpreting the outputs of such models and deploy them in the real-world applications.
Acknowledgement
This work is supported by a gift from Open Philanthropy, TPU Research Cloud (TRC) program, and Google Cloud Research Credits program.
References
[1] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari Morcos. Semdcdup: Data-efficient lcarning at web-scale through semantic deduplication. arXiv preprint 2333022303009540, 2023.
[2] Ccan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision, 2022.
10

---

[5] James Bradbury, Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne et al. Jax: composable transformations of python+ numpy programs. 2018.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot leareers In NeurlPS, 2020.
[7] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre vre-training to recognize long- -ail visual concepts. In CVPR, 2021.
[8] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, arXiv preprint arXiv: 1604.06174, 0674, 2016.
[9] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. ICMI, 2020.
[10] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[11] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212.07111 2022.
[12] Aakanksha Chowdhery, Sharan Narang, Jacoh Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. 2022.
[13] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune language models. arXiv preprint arXiv 14466, 2022.
[14] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? arXiv preprint arXiv:2204.11 134, 2022.
[15] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In CVPR, 2009.
[16] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
[17] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 18X1::1810.04805, 2018.
[18] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICIR, 2020.
[19] Samir Yitzhak Gadre, Gabriel Ilhareo, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 14108, 2023.
[20] Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao. Convmae: Masked convolution meets masked autoencoders. arXiv preprint arXiv:2205.0388 03892, 2022.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[22] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of ou dif-iitribbttion generalization. In ICCV, 2021.
[23] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[24] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip. July 2021.

---

[25] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[26] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeftrey Wu, and Dario Amodei. Sealing laws for neural language models. arXiv preprint urXiv:2001.0888
2020.
[27] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data, ICCV.
[28] Runze Li, Dahun Kim, Bir Bhanu, and Weicheng Kuo. Reclip: Resource-efficient clip by training with small images. arXiv preprint arXiv:2304.0000 06028, 2023.
[29] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofor, and Kaiming Hc. Scaling language-image pre-training via masking. In CVPR, 2023.
[30] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[31] Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jac Lec, and Chunyuan Li. Learning customized visual models with retrieval-augmented knowledge. arXiv preprint MXXi:2301100094 2023.
[32] Zhuang Liu, Hanzi Mao. Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, 2022.
[33] Tyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICI.R, 2017.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
[35] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
[36] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In ECCV, 2022.
[37] OpenAL. Gpt-4 technical report. 2023.
[38] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.0211 2022.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, perforerformncc deep learning library. In NewrIPS. 2019.
[40] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
[41] Filip Radenovie, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. arXiv preprint SxXiv:2301.022600 2023.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In /CML., 2021.
[43] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[44] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- textext transformer. Journal of Machine Learning Research, 2020.
12

---

[46] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[47] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[48] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyzz, Alex Salcianu, Mare van Zee. Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsyyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin- Shitin-Shepard, Noah Fiedel, Mark Omermick, Bremnan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203. Sifi:2203.7718,, 2022.
[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[50] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. In NeurIPS, 2022.
[51] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus, Learning visual representations with caption annotations. In ECCV, 2020.
[52] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b; An open large-scale datasct for training next generation image-text models. In NeurlPS, 2022.
[53] Christoph Schuhmann, Richard Veneu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Thco Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open datasct of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.00 14, 2021.
[54] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 23..:.233......9... 15389, 2023.
[55] Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, and Zehuan Yuan. Designing bert for convolutional networks: Sparse and hierarchieal masked modeling. arXiv preprint arXiv:2301.03580, 2023.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[57] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao, Clip-nerf: imagge driven manipulation of neural radiance fields. In CVPR, 2022.
[58] Haohan Wang, Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurlPS, 2019.
[59] Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b. arXiv preprint arXiv:2303. 12733, 2023.
[60] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. arXiv preprint arXiv:2301. 008808 2023.
[61] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling, arXiv preprint 2rx.0:2302.033/69 2023.
[62] Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint 1rXiv:2109.14000 2021.
[63] Hu Xu, Saining Xie, Po- Yao Huang. Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Cit: Curation in training for effeetive lasgon-language data. arXiv preprint m0Xiv:230..02244 2023.
[64] Jiarui Xu. Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- Vocabulary Panoptic Segmentation with Text-to-Imaa Image Diffusion Models. arXiv preprint 200.003333 2023.
13

---

[65] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-lext foundation models. arXiv preprini arXiv:2205.01911 2022.
[66] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li. Chunyuan Li, et al. Florence: new foundation model for computer vision. arXiv preprint 111v:2111.11432.22 2021.
[67] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- saiinnlanguage models behave like bags-of ofosds.-words, and what do about ICLR, 2022.
[68] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, arXiv preprint arXiv:2106.04 045600 2021.
[69] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1rX1::1110.0496, 2019.
[70] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
[71] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Xupeng Miao, Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In CVPR, 2022.
[72] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference.
[73] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:3304.06934 2023.
14

---

A Implementation Details
Emhed
Vision Transformer
Text Transformer
# params (M)
model
dim
layers
width
heads
layers
width
heads
vision
text
total
55
B/16
512
12
768
12
12
512
86
53
141
L/16
768
24
1024
768
303
109
414
H/14
1024
32
1280
1024
16
631
334
967
G/14
1280
48
1664
1280
1844
672
2516
Table 4: CLIP [42] model configurations used in our paper.
Config
Value
optimizer
AdamW [34]
optimizer momentum
(0.9, 0.95)
batch size base Ir minimal warm-up steps schedule
32768
8e-6
0
1600
cosine decay [33]
weight decay random crop area resize method color jitter [9] temperature init
0.2
(40, 100)
bi-linear
0.32
1/0.07 [24, 29]
Table 5: Pre-training hyper- sapper-prramters
A.1 Architectures
Our experimental results are based on specific model configurations shown in Tab. 4, following FLIP [29]. Our visual encoder architecture employs three different scales (S/16, B/16, and L/16) with the same patch size, allowing us to investigate the effect of scaling. In our CLIPA models, we employ vanilla ViT [18] with global average pooling as the visual encoder. The sine-cosine positional embeddings used in VIT [56]. As for text encoder, adopt the non-autoregressive Transformer [56, 29] and employ a WordPiece tokenizer [17]. which includes a "CLS" token for each input text. apply zero-padding to those input texts that shorter than the maximum token length of our model. For the ConvNeXt [32], we employ the same mode configuration described in [32]. and follow the setting in [24].
To ensure uniformity in the input length,
A.2 Hyper-parameters
Pre-training. Our CLIPA pre-training configuration outlined in Tab. 5. Notably, due to limited resources,
base learning rate of 8e-6 and smaller 32k batch size. In addition, apply a color jitter augmentation of strength 0.32 and probability 0.8, and a gray-scale augmentation of probability 0.2 [36,9 9, 10].
Fine-tuning. Following pre- ce-trrinning, conduct short- plort-period fine-tuning of the models using full-resolution images of size 224 x 224 and texts with a maximal length of 32. The fine-tuning process consists of 4000 steps with 800-step warm-up period. The base learning rate for fine-tuning set to 8c-7, while other parameters remain the same as those used during pre-training. Note that due to limited computation resources, our CLIPA-B/16 and CLIPA-L/16 7k batch size. The results different fine-tuning batch size are shown in Tab. 6. It can observed batch size of 32k.
fine-tuned with 8k and
that batch size of 8k already achieves competitive performance compared
We implement two codebases based on JAX [5] and Pytorch [39] respectively. Our JAX codebase built on Big Vision [4] and our pytorch code base mainly followed OpenCLIP [24]. Most of our experiments are conducted with TPU-V3, except that CLIPA-B/16 and CLIPA-L/16 in Tab. are conducted with A.00 GPUs
Implementation.
CLIPA-H/I4 and G/14. For comparison with previous state-of-the-arr models scaling experi- ments, we follow the approach in FLIP [29], using a 64k batch size for pre-training, and adjusting the warmup steps to 3200 to mitigate the unstable training of larger models. For fine-tuning at 224 resolution, apply random masking for image encoder with a 30% mask ratio to expedite the
15

---

Model
32k
164:
8k
4k
2k
CLIPA-L/16 ((17,T16)
68.1
67.8
67.7
67.1
66.8
Table Ablation on fine-tuning batch size.
Random mask
Resize
Block mask
Grid mask
2
Performance drop (%)
4
10
99
10
101
0
99
197
90
17
17
17
5.5
14.1
$1.4
31.4
5.5
Model size (1e7)
Model size (1e7)
Model size (1e7)
Model (1e7)
Figure 9: Model size vs. Performance drop. Different lines are for different total numbers of input
image tokens per sample.
Random mask
Resize
Block mask
Grid mask
65
65
60
610
60
60
Accuracy
000
100
000
05-
50
5/16
50-
6/16
50-
S/16
50
S/16
B/16
B/16
B/16
W166
I/16
I/16
1/16
1/16
35
85
65
45
2.9
Total pre-training tokens (le11)
Thtal pre-raining tokens (le11)
Total pre-trainng takens (lel1)
Total pre-traming tobens (tell)
Figure 10: Total number pre-training tokens vs. Accuracy.
training process and reduce memory costs. And we utilize 32k batch size with learning rate of 4e-7 and train with 512M training samples. In fine-tumin fine-tuning at 336 resolution, the mask ratio is at 40% for both models. We further reduce the base learning rate le-7 and train additional 128M samples with batch size of 16k. These two models are trained on 256-core TPU-V3 pod. We incorporate model-sharding strategy in our G/14 model, which is based on the T5X implementation [48]. Apart from this, we employ a distributed data-pan para-parlllel strategy.
A.3 Evaluation Setting
Our evaluation protocol is largely based on the original CLIP paper [42], and we employ the benchmarking tool provided by OpenCLIP [24]. To evaluate our model on ImageNet-Ik [15], we use 80 prompt templates for zero-shot testing. Following OpenCLIP [24], we resize the shorter side of the input images 256, and then perform center crop size 224 224. When a larger resolution of 336 is adopted, we directly resize the image into 336 x 336 without cropping.
16

---

model
GAP
color-
[9]
pytorch-impl. fine-tuning batch size
pre-train
fine-tune
Baseline
32k
58.3
66.3
32k
59.8
67.8 (+1..)
32k
58.5
67.1 (+0.8)
32k
59.9
68.1 (+1.8)
59.9
67.6 (+1.3)
67.8 (+1.5)
Table 7: Training details analysis. We use CLIPA-L/I6 (117, T16) model as baseline and report ImageNet-Ik [15] zero-shot top-1 performance. GAP: global average pooling in visual encoder. Pytorch-impl. we re-implement JAX [5] version and reproduce the results with Pytorch [39] on GPUs
B
More Results
In this section, we present more detailed results, including an alternative view of inverse scaling law, detailed ablation studies of training details and fine-tuning, and numeric results on ImageNet- 1k, which we used plot Fig. in the main text.
B.1 Alternative view of inverse scaling law
For improved representation, we offer two alternative interpretations of Fig. 4 in Fig. 9 and 10. In Fig. 9, we plot the model size on the x-axis, depicting fractions of token reduction as separate lines. In Fig. 10, the total pre-training tokens used the x-axis. In both perspectives, it's apparent that larger models demonstrate significantly smaller performance decrease when using fewer tokens for pre-training.
model
image Loken
# text Loken
data source
# seen sumples
total compute (xlel1)
IN-IK
CLIPA-L/16
36
8
LAION 400M
2.56B + 128M
0.5
69.3
LAION-400X
2.368 + 128M
0.8
72.8
CLIPA-H/14
36
LATON-2B
2.56B + 128M
0.8
74.1
LAION-2B
12.8B + 128M
77.9
Table 8: Scaling up CLIPA. Specifically, we explore scaling from the aspects of data. model, and
schedule. We pretrain the H/14 model with 36 image tokens (84 x 84) and text tokens; for fine- tuning, we use 256 (224 x 224) image tokens and 32 text tokens.
B.2 Ablation
Ablation on Training details We present comprehensive analysis of the training details employed to CLIPA. The results are summarized in Tab. 7. First, using global average pooling in ViT, instead of class token as in [24], leads to substantial improvement of ~ 11.5% in ImageNet- 1k zero-shot accuracy. Second, we also observe that incorporating stronger augmentation techniques [9] leads to a ~0.8% improvement. Together, they yield notable 1.8% improvement. It is also worth mentioning that to ensure a fair comparison, we switch to the widely-used OpenCLIP codebase [24], which is implemented in PyTorch [39]. Finally, to accommodate the limited GPU memory, employ a batch size of 7k for fine-tuning the CLIPA-L/16 model. Our experiments demonstrate that this adjustment results in only a marginal decrease in performance.
A Alltion scaling up. We next investigate the scaling behavior of CLIPA. Specifically, scaling efforts cover three aspects: model, data, and training schedule. The results are reported in Table
First. observe that scaling the model size from L/16 to H/14 boosts the performance from 69.3% to 72.8%. Furthermore, we note switching the training dataset from LAION-400M [53] to LAION-2B [52] yields another 1.3% improvement, suggesting the importance of data diversity. Lastly, by increasing the training schedule by factor of 5, resulting in a total of ~-1BB seen samples, achieve an impressive performance of 77.9%. We stress that this scaled version of CLIPA-H/I model readily outperforms its counterpart in FLIP [29] by 0.3% while requiring only 1/3 of the training budget.
These results confirm the efficiency and effectiveness of training CLIPA at scale. Next, we set this CLIPA-H -/14 with 77.9% performance our bascline for further ablation in the fine-tuning stage.
17

---

masking ratio
random
block
grid
case
masking ratio
resolution
# seen samples
training FLOPs
IN-1K
0%
17.3
17.9
77.9
baseline
0%
224"
128M
25%
78.2
78.0
77.9
(1)
50%
221"
128M
135.90
78.0
50%
77.7
77.5
77.6
(2)
30%
224"
512M
75%
76.2
74.3
76.2
(3)
30%
2242
640M
135. 9G
78.5
40%
336
610M
237
Table 9: Comparison of dif-
(5)
30%
224"
336
512M+128M
156.3G
79.1
ferent masking strategies. The results are obtained on on the LAION-2B dataset with H/14 model.
Table 10: Ablation on fine-tuning schedule and masking. In case
(5), we use 224 x 224 input with a masking ratio of 30% for the first 512M samples, and 336 336 input with masking ratio of 40% for the rest 128M samples.
S/16
B/16
1/16
Masking strategy
Masking
pre-train
pre-train
line-tune
pre-train
baseline
0.0%
197
56.7
64.2
69.2
randomn
50.0%:
54.7
55.5
61.9
62.6
68.3
68.5
grid
50.0%
99
53.9
54.6
62.5
62.8
68.3
68.5
block
50.0%
55.6
63.2
63.5
69.2
69.5
resize
160 x 160
101
54.0
56.0
62.2
63.6
67.8
69.0
random
75.0%
50
49.5
52.7
58.51
60.9
65.9
67.6
grid
75 53.00%
49.5
53.1
$7.9
60.7
65.4
67.3
block
75.0%
50
45.2
53.4
57.3
61.4
65.2
68.5
random
81.6%
55.3
grid
81.6%
37
N/A
N/A
N/A
N/A
N/A
N/A
block
81.6%
43.6
51.7
54.8
60.7
resize
96 96
48.3
53.9
57.0
62.1
63.8
68.1
random
91.8%
17
36.4
41.5
44.2
55,3
55.3
62.4
grid
91.8%
N/A
N/A
N/A
N/
N/A
N/A
block
91.8%
17
28.4
46.8
38.3
55.3
49.6
62.9
resizing
64 64
40.7
50.5
$1.0
59.9
58.3
66.2
Table 11: Scaling effect on reducing image tokens. We report top-1 zero-shot accuracy on ImageNet- Ik [15] classification. N/A: we adopt 50% and 75% masking ratio for grid mask, larger masking ratio is non-trivial. To ensure fair comparison, we keep the length of text tokens constant at 32.
S/16
B/16
L/16
Masking strategy
Image
Text
pre-train
fine-tune
pro-train
fine-tune
pre-train
fine-tune
muncation
112 112
32
50.1
54.7
59.0
62.9
63.1
random
112 112
32
50.0
54.8
59.1
62.6
65.3
68.6
block
112 x 112
32
50.4
54.6
59.1
62.9
65.2
68.7
syntax
62.6
tuncation
112 112
16
50.6
55.1
58.7
62.4
63.4
68.8
random
112 x 112
16
49.8
54.5
58.4
62.2
65.1
68.5
syntax
112 112
16
50.2
54.7
58.9
63.0
65.3
68.8
truncation
112 112
8
45.7
54.2
54.7
62.2
62.2
68.2
block
112 112
8
45.4
54.1
54.0
62.1
61.6
68.2
synlax
112 112
8
46.7
54,6
55.6
62.9
62.3
69.0
random
112 112
6
299
52.1
38.4
59.7
48.0
66.8
block
112 112
6
29.3
52.9
38.3
61.0
46.8
67.8
syntax
112 112
31.1
53.7
39.7
61.8
49.3
68.4
Iruncation
112 112
4
24.1
49.0
32.6
57.7
40.4
63.6
random
112 x 112
4
22.0
48.9
29.3
57.1
39.6
63.6
block
112
24.0
31.5
58.8
39.6
65.8
syntax
112 x 112
4
24.7
51.5
32,2
59.6
39.6
66.3
Table 12: Scaling effect on reducing text tokens. We report top-1 zero-shot accuracy on ImageNet-
1k [15] classification. Bold represents the best performance among different reducing strategies.
Ablation on fine-tuning schedule and masking. In addition to random masking, we hereby investi- gate how grid masking and block masking affect fine-tune tne-tuning performance. The results are reported in Table 9. Interestingly, compared to fine-tuning input tokens at the full resolution, we observe that 25% masked random fine-tuning and block fine-tuning all lead to slight performance improvement. With a larger masking ratio, all these masking strategies will lead to worse performance than full-resolution fine-tuning, but overall, random masking consistently yields stronger performance than the other two masking strategies.
18

---

VTT-5/16
Con-NeXI-T
VIT-R/16
Nou--e
Image size
pre train
fine tume
train
fine tume
pre train
fine tume
Irain
fine tune
224 x 221
56,7
26.7
64.2
64.0
160 x 160
54.0
56.0
55.5
56.2
62.2
63.6
63.0
63.6
112 x 112
50.1
54.7
52.6
55.3
59.0
62.9
61.0
63.1
96 96
48.3
53.9
51.0
54.6
57.0
62.1
59.2
62.2
64 x 61
40.7
50.5
44.9
51.1
51.0
59,9
54.6
60.0
Table 13: Comparison of ConvNeXt [32] and VIT [18]. We report top-1 zero-shot accuracy on ImageNet-1k [15] classification. ensure fair comparison, we keep the length of text tokens constant at 32 and only vary the visual backhones. The associated text encoders are specified in Tab. 4.
We next ablate different fine-tuning setups and summarize the results in Table 10. We choose 30% masked random fine-tuning as the default strategy, as it leads slight performance improvement (+0.1%) and enables a 1.3x speedup of the fine-tuning process. Furthermore, adopting 4x fine- tuning schedule results in an additional improvement 0.6%. However, we empirically find that further increasing the fine-tuning schedule does not lead to any substantial performance gains.
Following [24], we also investigate progressively fine-tuning with large image resolutions. Initially, for the first 512 million samples, we fine-tune the model using 224 x 224 input size with a masking ratio of 30%; subsequently, for the remaining 128 million samples, we adopt larger 336 336 input size with masking ratio of 40% and smaller learning rate. As shown in the last row of Table 10, i.e. case (5), progressive fine-tume tin-tuning results in a slight performance improvement of 0.2% compared to direct tun-nuning with 336 x 336 input size and meanwhile achieving a notable 1.5x speedup of the fine-tuning process.
model
NegCLIP
VG-Relation
VG-Attribute
COCO-Order
Orickr30k-Order
OpenCLIP. OpenCLIP-B/16
44.7
(8339)))
(+9.6)
87.6 (+45.8)
89.1 (+43.8)
CLIPA-B/I6
43.8
$7.1
37,8
39.1
CLIPA-B/16
V
78.5
(+34.7)
(+10.9)
86.1 (+48.3)
87.9 (+48.8)
Table 14: Comparison on ARO benchmark.
B.3 ImageNet-Ik
Image. For reference, Tab. 11 presents the numerical zero-shot ImageNet-Ik top-1 accuracy of Fig. with various token length reduction strategies. We can see that fine-tuning plays a crucial role with reduced input token length during pre-training, by comparing the performance of pre-trained and fine-tuned models. For instance, fine-tuning pre-trained models with only 17 tokens (the last four rows in Tab. 11) leads to significant performance gains of 18.4%, 17.0% % and 18.6% across S/16, B/16, and L/16 scales, respectively, for block masking.
Text. For reference, Tab. 12 presents the numerical zero-shot ImageNet- Ik top-1 accuracy of Fig. 7 with various token length reduction strategies. We standardize the visual input size to 112 x 112 pixels for all models and vary only the text input during gre-training. Our fine-tuning procedure follows the same approach as that described for the image input. To ensure fair comparison, employ the same masking strategy during fine-tuning as used during pre-training when comparing different masking strategies. Note that the pre-training performance is noticeably lower for input texts with a length smaller than 8. This is because the prompt templates we used for evaluation are often longer than text length of 8. However, after fine-tuning the model with maximum length of 32, the models performances are significantly improved.
ConvNeXt [32]. In Tab. 13, we compare the performance of different visual backbones. VIT [18] and ConvNeXt [32]. Notably, we observe that when comparing pre-training results, ConvNeXt outperforms ViT with smaller input sizes, suggesting that CNNs may exhibit greater robustness with respect to scale. For instance, at an input size of 64 x 64, ConvNeXt-B outperforms ViT-B by approximately 3.5%. However, after fine-tuning, we observe that the performance gap between the two models narrows considerably across all scales.
19

---

B.4 Zero-shot retrieval and robustness
Text. For reference, we also report the performance of zero-shot image/text retrieval on the COCO dataset [30]and zero-shot robustness on the ImageNet- V2 [47], ImageNet-R [22], ImageNet-A [23]. and ImageNet-Sken Seethh [58] dataset in Tab. 15 when varying input text token lengths.
Bri6
mealcing stnilags
Inaac
Lost
LN-Ik
15-A
IN-V2
IN-A
IS-V2
IN-A
cenkon
56.8
25.3
55.
biok
112
17
50.6
113
1055
51
51.5
55.
98.2
182
112
32
54.9
40.5
142
63.4
41.1
30.9
49.2
62.
54.5
25.6
73.5
45.3
31.2
55.3
66.3
60.5
35.3
50.3
55.4
41.
99.7
112
55.1
07.2
14.3
47.8
54.3
25.7
37.1
254
413
59.
caskon
112
56.3
40.3
56.8
28.3
48.00
31.2
28.9
4017
59.7
book
102 x 110
16
50.5
65.9
14.5
02.6
40.3
51.1
49.1
63.3
55.3
25.6
13.3
45.6
37.1
55.0
68.7
00.0
37.2
50.3
55.7
41.2
59.6
Jontas
182 X 112
18
54.7
42.4
13.4
62.1
40.5
34.5
25.5
0331
35.4
25.5
23.3
4933
31.1
15.3
68.5
61.4
37.6
B6L3
25.6
41.1
28.6
183.00
112
56.3
06.1
14.1
65.5
10.1
78.1
47.3
63.3
51.9
26.7
76.1
48.3
35.5
53.1
66.3
61.0
37.0
1013
55.
1966
96.7
Deskon
182 112
55.3
65.6
13.7
62.6
19.1
73.6
47.1
63.5
51.9
26.8
71A
41.2
31.6
12.6
K7.
99-1
30.2
18.3
56.1
77.8
567
block
182 112
56.1
13.8
02.5
40.0
47.9
62.1
54.2
25.0
73.4
00.3
37.1
50.5
55.8
40.4
57.5
102 112
54.6
13.5
63.5
47.7
62.9
54.7
25.7
73.4
61.5
39.1
56.
36.7
57.7
112 112
52.9
13.3
63.6
39.3
60.8
51.5
24.2
73.2
59.5
35.3
5L.
58.2
54.9
61.
18.0
11.9
23.1
5L.5
36.9
513
26.1
55.1
51.2
53.7
54.2
01.0
563
5011
21.3
11A
Table 15: Zero-shot image/text retrieval and robustness, We also study the scaling effect on text tokens on image/text retrieval tasks and robustness benchmarks.
20


Reviews:
[{"review_id": "PZXBxeFhoj", "reviewer": "Reviewer_Mhr1", "summary": "The key finding in this paper is that, the larger the model is, the shorter the token sequence is required to maintain a certain level of performance. Based on this finding, this paper proposes to train larger model with shorter sequences, which is more computationally favorable than training smaller models with sequences of full length. As a result, one can achieve 69.3% top-1 zero shot image accuracy with just eight A100 GPUs in ~4 days.\n\n", "questions": "see above", "limitations": "I would not say this is the limitation, but it pushes me to wonder how general this inverse scaling law is? e.g., would it be easily applied to other tasks?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The finding about the inverse scaling law for CLIP is interesting. Given that CLIP is such a significant way of training vision language models, the inverse scaling law may yield high impact in this direction as well.\n\n- The writing is overall good except some minor points (see in the next section). The visualization in Figure 2 and 3 well demonstrate the shortening strategies used, making it easier to understand for readers who can only afford a quick scan. The results presented in Figure 4, 5, amd 6 are clear, and easy to follow.\n\n- The experimental results are strong. As shown in Table 3, one can train the same arch without much drop (e.g., 2-4% drop) but saving 10x-20x GPU hours. This is can be very important for academic labs where resources are not huge.\n\n- The figure 7 shows that, when given a fixed number of computation budget, you can achieve better accuracy with larger models with this inverse scaling laws applied.", "weaknesses": "I did not see major flaws of this paper, but with some minor comments:\n\n- firstly, I think this paper misses a subsection or paragraph in Related works, which can talk about similar findings in language models. To my best knowledge, it has been well known that, larger models can be trained with less steps/epochs/durations compared with smaller models. One example is the paper \"Scaling Laws for Neural Language Models\". So I think it would be nice to give readers a full spectrum of knowledge by including a series of such related work in the large language domains, and I would not think including them will make this paper less innovative, rather this paper becomes more complete.\n\n- this paper claims multiple times that CLIP is the first foundation model that connects images and text. I am less comfortable about this claim, since there are a bunch of image-text relevant work before CLIP, e.g. such as image captioning model. Having not trained with modern large models does not mean that they are not foundational or they could not have been foundational models if they were trained with large models. This is a minor point, but would love to see more rigorous and nice claim.\n\n- Why there is performance improvement for block mask in Figure 4, and similarly why this is happening fro Figure 5 too. And it seems there is in consistency between different tasks in terms of which strategy can improve or not?\n\n- Why it improves a lot for ImageNet-A in Figure 6, while not the case for other datasets/tasks?\n\n- in the paragraph of line 204, it says syntax masking even enhances the performance for CLIP. I wonder if this is mainly because it keeps nouns, which are more informative for tasks relevant to object recognition. What about some other tasks that focuses on predicting/modeling relationships between objects? Would syntax masking still effective, or would still text masking useful at all?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for the detailed comments and the appreciation of our work. We address the concerns below\n\n### Q1: Missed related works of scaling law in language models\nThanks for this suggestion! We completely agree that discussing these related works will make this paper more complete, as we share a similar message — that using larger models doesn't necessarily entail a proportionate increase in training computation. We will fix this in the next version.\n\n### Q2: Rephrase the statement of CLIP\nThanks for this suggestion. We will add a detailed discussion about the prior efforts on image-text learning and clearly posit CLIP as one of the pioneering works that push this research direction at scale.\n\n### Q3 & Q4: Performance improvement with masking & Significant improvement on ImageNet-A\nIntuitively, learning with more tokens comes with more information and generally should lead to better performance. But regarding the interesting observation in Figures 4 & 5 & 6 that learning with fewer tokens sometimes leads to improved performance, one possible conjecture is that masking at a low ratio might act as a form of regularization — by filtering out potentially noisy or redundant information, it could help the model to focus more on key features, thereby enhancing representation learning. We believe a more rigorous and systematic investigation of this phenomenon would offer valuable insights and leave it as a future work.\n\nOverall, these findings, while intriguing in their own right, well support the core message of this paper — that the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training.\n\n### Q5: Text masking in objects relationship modeling tasks\nGood question! Our intuition agrees that nouns are particularly relevant or useful for object recognition tasks, therefore the learned representation may not be directly effective for tasks that require a more complex understanding of relationships between objects. However, it is essential to note that our training pipeline contains two stages — the reduced-length input is only applied during pre-training to accelerate CLIP training, and full-length input is used during fine-tuning to ensure that the models access all available information in the training data. This two-stage process enables nuanced modeling of relationships between objects. We will add this discussion in the next version.\n\n### Q6: Generalizability of inverse scaling law\nWe are optimistic about the potential for this inverse scaling law to extend to other image-text learning tasks. This optimism stems from two main observations: 1) Image and text data frequently include redundant information, and 2) Larger models typically possess enhanced representation learning capabilities. We are excited to keep exploring this direction.\n"}, {"review_id": "qNyYrfbLYD", "reviewer": "Reviewer_UF9M", "summary": "This work studies the impact of token length (image and text) on the quality of image-language pretraining in CLIP-style models. Crucially, the authors find that as the model size (both image and language encoders) are scaled up, the drop in accuracy owing to a smaller input token length keeps on decreasing. This is pitted as the inverse scaling law. A smaller input token size enables training significantly faster, achieving nearly 15x speed up over the conventional training protocols, and also improving over the gains observed in prior work, FLIP.", "questions": "1. What is the performance of CLIPA models without fine tuning\n2. How was the fine tuning duration and learning rate chosen?\n3. How were parameters for pretraining chosen?\n4. What are the embedding dimensions of the models when the token lengths are changes (see Figure 1)", "limitations": "Discussed in Weaknesses", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The work provides a comprehensive analysis of various token reduction strategies for both image and text and also proposes new ones.\n2. Analysis of different scales of models, across different masking strategies is really helpful to convert a previously under-explored finding into a consistent phenomenon that can be useful for future research.\n3. The work makes large model training accessible by suggesting an alternate training paradigm of pre-train and then fine-tune that enables CLIPa training at a much lower cost than before (15x speedup).", "weaknesses": "1. I find Figure 1 confusing. What is the relation between the encoder input token length and the encoder output \"image/text embedding\" dimension. This should be discussed in more detail in the paper. Also, since the whole work is about token length reduction, I think it is important for the authors to discuss how images are converted into patches and then into tokens and how this calculation is done.\n2. I do not feel very confident about the results in the paper because the parameter choices appear to be heavily tuned in order to get good performance for the chosen strategy of resizing. For instance, the authors use a base learning rate of 8e-6 in the main training stage and 4e-7 for fine-tuning---these are non standard parameters and a discussion on parameter tuning is important here.\n3. Similarly, in line 147 the authors write that \"one exception that a larger learning rate of 8e-7 is utilized during fine-tuning for S/16 and B/16.\" Such opportunities were not given to baselines such as FLIP. Further, special tricks and augmentations were used for H-14 making it unclear how the results will translate further.\n4. Scaling Law: Since this paper advertises a scaling law, the graphs should have model size on x axis and the various fractions of token reduction as individual lines. Moreover, there should be more points on the \"new\" x axis to claim a scaling law.\n5. Why are there no graph for results corresponding to Table 1.\n6. Evaluations: Model performs significantly worse on text retrieval tasks. I think the authors should do a more comprehensive evaluation of CLIPA on various benchmarks such as VTAB (check datacomp paper for current best practices).\n7. Table 3 should compare with FLIP and other masking strategies\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for the detailed comments. We address the concerns below:\n\n### Q1: Figure 1 confusing & Discussion about patchifying\nSorry for the confusion about Figure 1. There is a typo in the figure; the dimension of the encoder output solely depends on the encoder size (i.e., a larger encoder size leads to a higher output dimension) and is unrelated to the input token length. We will fix it.\n\nRegarding patchifying process, we follow standard ViT practice to partition an image into $p×p$ (i.e., p=16 or 14 in our experiments) non-overlapping patches and project each patch into a token. To reduce the token length, we apply techniques such as random, block, or grid masking to remove tokens. When resizing, the input size diminishes, but the patchify size remains consistent, resulting in a reduced token length.  We will clarify it in the next version.\n\n\n### Q2 & Q3: Heavy & Unfair parameter tuning\nWe appreciate your concern about potential heavy or unfair parameter tuning. Nevertheless, we wish to clarify that our approach is not characterized by such practices. Our training setup predominantly aligns with FLIP's, with the only exception being the use of a slightly higher learning rate. It's also important to emphasize that our random masking baseline corresponds to FLIP and is trained using this new learning rate configuration. As our analysis in Section 3 indicates, random masking, as implemented in FLIP, is not the optimal solution for reducing token length in CLIP training. \n\nFurthermore, we emphasize that the key contribution of this paper is the discovery of the inverse scaling law in CLIP training. We firmly believe that this finding is obtained under a fair experiment setup.\n\n### Q4: Scaling law figure drawing\nThanks for the suggestion. Please see this new figure ‘with model size on the x-axis and the various fractions of token reduction as individual lines’ in the rebuttal file (Figure 2). We will add it to the next version to clarify the observed scaling law better.\n\n### Q5: Graph for Table 1\nWe have included this alternative view for Table 1 in the rebuttal file (Figure 1) and will include them in the revision.\n\n### Q6: A more comprehensive evaluation\nThanks for your suggestion. As requested, we compare OpenCLIP, FLIP, and our CLIPA models on VTAB benchmarks. The results are included in the rebuttal file, as shown in Table 1. As can be seen, on this highly diverse and challenging set of vision tasks, our CLIPA still achieves comparable or even superior performances but with significantly less training cost, demonstrating our approach's good generalizability. We will include these results in the next version.\n\n### Q7: Include other masking strategies in Table 3\nThe detailed comparisons among different masking strategies (including FLIP/random masking) are shown in section 4 (Figures 4-6). Given these findings, we made the decision to focus on the most effective strategies (i.e., resizing for image, syntax masking for text) when scaling to large models like H/14. We will make it clear in the next version.\n\n### Q8: CLIPA performance without fine-tuning\nWe reported the performances of pre-trained CLIPA models in the supplemental material shown in Tables 4, 5 & 6. We will make it clear in the next version.\n\n### Q9 & Q10: Choosing parameters for fine-tuning & pre-training\nOur pre-training and fine-tuning duration exactly follows the FLIP setup (in pre-training, 6.4 epochs for ablation and 32 epochs for scaling; 0.32 epochs in fine-tuning). The learning rates, as answered in Q2&Q3, are slightly larger than the FLIP choices. We will make it clearer in the next version.\n\n\n\n"}, {"review_id": "Oebalddo48", "reviewer": "Reviewer_KzVG", "summary": "This paper is about increasing the training efficiency of vision-language models such as CLIP. These models can serve as a foundation for many vision applications and are therefore very useful. However, they are typically (pre-)trained on large datasets for many steps, which makes research involving training such models costly. This motivates efforts to reduce the training cost.\n\nThe paper focuses on the sequence length of both the image encoder and the text encoder in CLIP-style VLMs, which is a key determinant of model compute and memory requirements. The paper evaluates random token dropping, dropping tokens in grids or blocks, and reducing the image resolution. The paper finds that reducing image resolution performs best based on zero-shot ImageNet classification performance.\n\nFor the text encoder, the paper finds that dropping text tokens such that nouns are preferentially kept and other words are dropped yields the best performance, again on zero-shot ImageNet classification.\n\nThe paper also sweeps across different model sizes and finds that larger models can tolerate larger reductions in sequence length on both the image and text encoder.\n\nWith the proposed sequence length reductions, it is possible to train models matching or exceeding the performance of OpenCLIP (but not original OpenAI CLIP) on image classification and image/text retrieval in 10-20x less training time.\n", "questions": "Further suggestions:\n* Please provide Table 1 data in graphical form like Figure 4.\n* Table 3 should include inference FLOPs to be able to compare models.\n\nTypos:\n* L170: For instance, For instance\n* L208: Double ampersand", "limitations": "The paper does not address the concern that the limited evaluation of the proposed model may hide model deficiencies compared to the original CLIP. This should be addressed in the rebuttal.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The work is well motivated. Reducing the cost of pretraining foundation models has broad impact.\n2. The reduction in training time is significant and could make research on training VLMs easier (although it comes with caveats, see below).\n3. The presented experiments are clear and and well-structured.", "weaknesses": "1. **Incomplete evaluation:** The paper relies on zero-shot classification and retrieval to evaluate the proposed model against the original CLIP and OpenCLIP. This covers only a small part of the tasks that CLIP is typically used for. The original CLIP paper also evaluates representation quality with linear probing and finetuning, on a much wider range of tasks and image types. These evaluations may be more informative about general representation quality than zero-shot image classification. In particular, approaches such as reducing the input resolution and dropping non-noun words may work for classification, where there is a single big object in the image and the text usually consists of a single noun, but might harm more structured or fine-grained downstream applications. This should at least be discussed and ideally tested, e.g. with linear probing and finetuning, and a wider range of tasks, ideally including structured tasks such as segmentation or detection.\n\n2. **Inaccurate and unnecessarily polemic presentation:** The paper makes several inaccurate statements with the apparent motivation of drawing attention. These inaccuracies distract from the content:\n    1. **Inverse scaling law:** Despite the title, the paper does not propose a quantitative scaling law, and there is nothing “inverse” about the presented scaling results. Figure 7 shows that as the compute budget increases, the optimal model size grows. This is the same result first presented in Figure 2 of https://arxiv.org/pdf/2001.08361.pdf and confirmed in many works since. The observation that larger models tolerate larger reductions in sequence length neither makes the scaling “inverse” nor is it a surprising result, since bigger models are known to be more sample efficient (https://arxiv.org/pdf/2001.08361.pdf, https://arxiv.org/pdf/2106.04560.pdf), i.e. they learn faster from less data than smaller models. I suggest rephrasing the title to avoid confusion.\n\n    2. **CLIP restricts accessibility:** The abstract and introduction suggest that foundation models such as CLIP are “restricting accessibility to a small group of researchers and technology companies”. This seriously misrepresents the impact these models had on low-resource research. In fact, foundation models like CLIP made it possible to train a model once and then apply it zero-shot, or with cheap fine-tuning, to many different tasks. Sharing pre-trained foundation models therefore has been a huge gift to low-resource research, since complex research questions can now be explored with a pretrained CLIP model, often without having to do any model training. Portraying such models as hindering low-resource research is inaccurate and unfair towards the researchers who develop and release them. I suggest the authors revise these statements.\n\n    3. **Academic resources:** Another questionable premise is that of “academic” resources (e.g. in the title of Section 5 and the name of the proposed method). There are academic labs with large resources and industry labs with low resources. In addition, research labs in industry that publish their work can reasonably be considered to be part of academia (for reference, Wiktionary defines “academia” as “The scientific and cultural community engaged in higher education and research, taken as a whole.”). Equating “academic” with  “low resource” is therefore inaccurate and reinforces stereotypes. Please simply talk about “few/low resources” instead of “academic resources.”\n\nIn summary, additional evaluation and major revision of the presentation are necessary before publication at NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for the detailed comments. We address the concerns below:\n\n### Q1: Incomplete evaluation\nThanks for raising this concern! \n\nFirstly, it is essential to note that our training pipeline contains two stages — the reduced-length input is only applied during pre-training to enable fast CLIP training, and full-length input is used during fine-tuning to ensure that models learn from the complete information available in the training data. \n\nSecondly, as suggested by Reviewer UF9M, for a wider range of evaluation, we have included the results on VTAB benchmarks in the rebuttal file. The competitive performance of our CLIPA models on this diverse and comprehensive benchmark demonstrates the good generalizability of CLIPA. \n\nLastly, we argue that, compared to other learning paradigms, the biggest advantage of CLIP lies in its extraordinary zero-shot recognition ability. This is also highlighted in both the official CLIP repo and the popular OpenCLIP repo. Our work follows this common practice and our assessment of models’ zero-shot ability on a wide range of datasets demonstrates the efficiency and the effectiveness of our CLIPA.\n\nWe will add these clarifications in the next version.\n\n\n### Q2: Inverse scaling law\nWe first stress that our observation indeed suggests an inverse relationship between the model size and the length of image/text tokens in CLIP training — that the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training.\n\nSecondly, it's crucial to distinguish our findings from the scaling law in [1], though seemingly subtle. The sample efficiency in [1] highlights that in language learning, larger models need fewer samples to achieve the same performance as the smaller models. This concept could be applied to CLIP training (e.g., L/16 may need ~1/3  total input samples to reach the same performance as S/16), but this is NOT what we discussed/explored in this work. Instead, we argue that, compared to small models, we could drop more information **per training sample** to train large models (e.g., for a performance drop threshold of ~1%, S/16 needs 160x160 image input, but L/16 only needs 96x96 image input). \n\nTherefore, while both ours and [1] demonstrated that large models could be trained more efficiently, the specific scenarios and the underlying rationale are distinct, i.e., [1]’s token efficiency is realized by using few training samples (i.e., there is no token drop in each training sample), but our token efficiency is realized by using fewer tokens in each training sample (i.e., total training samples keep the same). To our knowledge, this specific observation is novel and unexplored in prior works.\n\nWe will make these points clear in the next version.\n\n\n\n\n### Q3: CLIP accessibility statement inappropriate\nSorry for the confusion. First, we agree that, by building upon open-sourced CLIP models, simple zero-shot applications or cheap fine-tuning of CLIP have made significant contributions to the field and led to many fruitful research outcomes. \n\nHowever, our comments on accessibility were specifically targeted at the training aspect of CLIP. While the release of the OpenCLIP codebase and the LAION datasets (400M and 5B) has indeed empowered researchers to train their own CLIP models, the substantial training cost associated with the large-scale model and data remains a significant barrier to further exploration. \n\nOur intention was to highlight this aspect of the accessibility challenge, as we believe that there is still ample room for community-driven innovation in improving or even completely revamping the image-text learning paradigm. CLIP's current form, though powerful, is likely not the final solution.\n\nOur discovery of the inverse scaling law addresses this challenge by enabling CLIP training with significantly reduced costs and faster research cycles. By making training more feasible, we hope to stimulate greater participation and creativity in the field.\n\nWe will make this point clearer in the next version.\n\n\n\n### Q4: Academic resources statement inappropriate\nThanks for the suggestion. We will revise our language to more precisely reflect our focus on low-resource scenarios (e.g., a standard 8-GPU server).\n\n\n**References**\n\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).\n"}, {"review_id": "8Ii9hPjzLK", "reviewer": "Reviewer_TeGF", "summary": "Different methods to reduce the number of tokens entering the image or text encoders when training a CLIP model are compared.\nIn doing this the authors discover that larger models are able to learn faster from fewer tokens per example than smaller models; this is the inverse scaling law in the title.\n\nModels are compared keeping the following variables constant:\n\n* Training set: LAION-400M\n* Total pre-training examples: 6.4 epochs\n    * Tokens/example is **not** constant so tokens/epoch will be vary accordingly\n* Total fine-tuning examples: 0.36 epochs\n* Learning rate: 8e-6 pre-training and 4e-7 fine-tuning\n    * However, this is not kept constant for S/16 and B/16 models in scaling experiments\n\nThe authors investigate alternative strategies to reduce tokens/example based on FLIP, designing them based on the attributes of images (grid masking: \"preserving one patch in each 2x2 grid window\") or attributes of text (syntax masking preferentially retains nouns). They then investigate the effect of scaling while reducing tokens/example.\n\nUsing the discovery that larger models learn faster from fewer tokens the authors are able to train a CLIP Academic (CLIPA) (I can't find where this is defined in the paper but I think this is what it is?) on LAION-2B using 1/15 the compute budget of OpenCLIP at the H/14 scale and achieve comparable performance.", "questions": "What is the formal definition of a scaling law this paper is applying, such that this demonstrates an inverse scaling law? I'm not sure if the field has one, it seems to be \"a straight line predicting performance on a y-axis and training cost on a logarithmic x-axis\".\n\nHave I misunderstood the role of the masking methods to reduce tokens/example? I don't think I have but my entire review depends on it.", "limitations": "Comparing this to a [Chinchilla][] scaling law framework it may be that CLIP models are typically trained with more tokens than necessary, in which case we would expect to observe larger models perform better than smaller models regardless of the reduction in tokens. This could be investigated in a way analogous to the Chinchilla paper, by varying the total number of training tokens seen be the model for different model scales and aiming to see the IsoFLOP curves shown in Figure 3 of the Chinchilla paper. However, this would be a computationally expensive experiment to run. Perhaps the authors can come up with a cheaper way?\n\n[chinchilla]: https://arxiv.org/abs/2203.15556", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The experiments in this paper are comprehensive and follow a reasonable thread of investigation.\nThe authors define clearly what they aim to investigate, then show complete results of that investigation, then apply those results to a pressing problem in the field.\n\nThe comparison of different methods to reduce tokens/example is valuable and the methods chosen for comparison seem like good choices. The results are presented in full in Figures 5 and 6 and Table 1.\n\nThe training results are striking and of obvious importance to the community. CLIP was a major step in the field of deep learning so speeding up training is valuable to increase access and to speed up experimental iteration on these architectures. The speedup of 16x is more than an order of magnitude.\n\nThis paper builds on the results of FLIP by making a useful observation on the effect of model scaling on token/example reduction strategies.", "weaknesses": "Figure 1 is difficult to parse because the number of tokens in the top row could indicate entries in the contrastive matrix or it could indicate the number of tokens at the input to the image/text encoders. From reading the paper, I think it is the latter but it would be helpful if this was immediately obvious. If this mirrored the design of Figure 1 in the original CLIP paper I think it would be very clear.\n\nThe results of this paper are quite different if we view epochs as being a measure of tokens or of steps. I think that it is a measure of steps because the alternative would be more difficult to code but I think it would be worth making clear so that's what I wrote in the summary above. I think to make this extremely clear and better state the scaling law there should be a figure comparing log total training tokens to loss or accuracy. From Figure 4 I think this would show increasing accuracy with the number of tokens processed by the network during pre-training.\n\nIt is not clear in Section 5 exactly which methods are applied in CLIPA training to reduce tokens/example, from the methods proposed in previous sections. It would be useful for this to be made clear, even if it is stated in prior sections or appendices, because the results hinge on this.\n\nThe speedup quoted in Section 6 and in line 51 compares to OpenCLIP but Table 4 shows similar comparison to the work this is most directly based on, FLIP, was trained with only 5x the compute. This is a more fair comparison and deserves to be stated instead of the alternative unless you qualify that it is 16x cheaper than training without doing anything to reduce tokens/example.\n\nSmaller issues:\n\n* In the abstract the bold results at the end are impressive only if one knows the training time of OpenCLIP but it might be more reasonable to assume that most readers won't\n* Lines 33-36 describe the idea of a scaling law in parameters but neglect scaling laws in tokens or compute etc, it might be valuable to be more precise\n* The text in Figure 5 x-axes is very small, could it be made more legible somehow?\n* I think Figure 7 shows the effect of the tokens/example reduction strategies on GFLOPS required for training but the caption does not explain what is happening in this Figure. Is GLOPS the number of flops per example, so the x-axis would show the total training cost?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for the detailed comments and the appreciation of our work. We address the concerns below:\n\n### Q1: Figure 1 difficult to parse\nThanks for raising the concern! The number of tokens in the top row indeed indicates the number of input tokens. We will update Figure 1 to make it clearer in the next version. \n\n### Q2: Epochs as a measure of steps or tokens\nThanks for this suggestion! As requested, we've included an alternative view in the rebuttal (Figure 3) that compares total pre-training tokens to accuracy, and will include it in the next version.\n\n### Q3: Token reduction strategy used in CLIPA\nThanks for the suggestion! Image resizing is used in all CLIPA training. As for text masking, it depends on model size. Specifically, for small models (S/16, B/16 and L/16), as the text encoder incurs only a small portion of the total compute, we follow FLIP’s practice and opt for truncation. However, when scaling to H/14 with 70x70 image input, we note reducing the text token from 32 to 8 results in a substantial reduction of ~40% in compute requirements. We select syntax masking for CLIPA H/14 due to its strong performance. We will make this setup clear in the next version.\n\n### Q4: Speed comparison with FLIP in section 6 \nIndeed, FLIP H/14 was only trained with ~6x compute compared to our CLIPA-H/14 model. This ~15x compute result is based on the OpenCLIP training with full-length input. We will make this speed comparison clearer in the revision.\n\n### Q5: Bold results in the abstract\nThanks for the suggestion! We will include background information about the original CLIP training cost in the abstract to better aid readers in comprehending the significance of the results presented.\n\n ### Q6: Line 33-36 a scaling law in parameters\nThanks! We will rephrase our statement about the scaling law to make it more precise and accurate. \n\n ### Q7: Text in Figure 5 axes too small\nThanks! We will increase the font size in Figure 5 axes for better presentation in the next version.\n\n ### Q8: Figure 7 caption\nYes, GFLOPS is the number of flops per example, and the x-axis of Figure 7 shows the total training cost. We will update the caption to make it clearer in the revision.\n\n ### Q9: Definition of the inverse scaling law\nThe common scaling law associates more compute and larger model scale with better performance [1]. The inverse scaling law reveals that larger image/text encoders allow for the use of shorter image/text token sequences in CLIP training. The 'inverse' naming refers to this negative correlation.\n\n ### Q10: The role of the masking methods\nYou understand it correctly! Appropriate input token reduction greatly accelerates CLIP training at little-to-no performance drop. This is the main point of this work.\n\n ### Q11: Token Redundancy\nIntuitively, we agree with the idea that the redundant information in the image-text pairs is what makes the token reduction strategy work. This is evidenced by our results in Figure 7. The larger models have stronger representation learning capacity, and thus can learn better with input of reduced-token version, surpassing the smaller models trained with full-size input by a significant margin. Further evidence lies in our CLIPA-H/14 model (I36, T8), which achieves performance comparable to OpenCLIP-H/14 using only about 1/15 of the total number of tokens.  We will add a related discussion in the next version.\n\n**References**\n\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).\n\n"}, {"review_id": "eL79XcZxQi", "reviewer": "Reviewer_Depv", "summary": "This work explores how to increase the training speed of CLIP in order to make it possible to train on academic resources. They examine how to best compress both the visual and text information given a limited token budget. They do this across multiple CLIP sizes and find that as the size of CLIP increases, it sees less of a drop for fewer tokens which they deem the \"Inverse Scaling Law\".", "questions": "- How do you see your work in relation to the concurrent RECLIP (https://arxiv.org/abs/2304.06028)?\n- What do you see as the main limitations of this work?\n- [minor note] typo in line 208 (double ampersands)", "limitations": "In future versions I would recommend a section for limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a very useful line of work, as I believe there is still a lot of research to be done in the Language-Vision space and the compute requirements of CLIP is a huge barrier. Allowing CLIP training on an academic resource budget is valuable to the greater resource community. \n\nIt is presented in an easily understood way. I especially Like figure 7, which summarizes their results nicely. ", "weaknesses": "My main concern is novelty. Their findings are very similar to that in \"RECLIP: Resource-efficient CLIP by Training with Small Images\" which also found that images could be sized down and text could be truncated without major accuracy loss, with the goal of increasing training accessibility. This paper essentially takes the main findings of this paper, and provides extensive ablations for a series of design choices (such has the best way to compress the images/text), but ultimately comes to a very similar conclusion. However, as RECLIP, was only on ArXiv 2 months ago, I believe it should be considered as concurrent work. I would however ask that the authors include a discussion of this work in their next version.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for the detailed comments and the appreciation of our work. We address the concerns below:\n\n ### Q1: Relation to RECLIP\nThanks for bringing up this concurrent work! We totally agree that a discussion of RECLIP [1] will make this work more complete, and we will do so in the next version. Our work does share some similarities with RECLIP. For example, both aim to ease the computational burden of CLIP training. And both propose image resizing as an effective token length reduction strategy. However, we would like to highlight our unique contributions as follows. \n\nBeyond RECLIP, our work takes one step further, aiming to investigate how the performance of different model scales responds to reduced-length input.  By exploring eight image/text token reduction strategies, we uncover the inverse scaling law, a novel observation that serves as a vital guide for accelerating CLIP training in general. Furthermore, we extensively validate the effectiveness of our approach with a wide range of compute budgets (ranging from 18 to 370 GPU days), model scales (from S/16 to H/14), training datasets (LAION400M, LAION2B, and DataComp-1B), evaluation benchmarks (Robustness, Retrieval, and VTAB), demonstrating the well-established generalizability of our work.\n\n\n\n ### Q2: Main limitation\nOur work's primary limitation is that we only focus on the inverse scaling law in CLIP training. While this has led to promising insights, we have yet to explore its applicability to other image-text learning tasks. Given that 1) both image and text data often contain (a lot) redundant information; and 2) larger models have stronger representation learning ability, we believe future endeavors in this direction to be potentially fruitful. This exciting direction for further exploration is left as future work.\n\n\n ### Q3: Typo\nThanks for pointing out the typo! We will correct it and further polish the writing of this paper.\n\n\n**References**\n\n [1] Li, Runze, et al. \"RECLIP: Resource-efficient CLIP by Training with Small Images.\" arXiv preprint arXiv:2304.06028 (2023)."}]

Author rebuttal:
### Global Response:

We would like to first thank all the reviewers for their invaluable comments and recognition of the interest and value in our discovery of the inverse scaling law in CLIP training (Reviewer Depv, TeGF, Mhr1), enabling the potential to improve accessibility to large models and significantly reduce training costs (Reviewer UF9M, KzVG). We also value the reviewers’ positive feedback regarding the clear presentation of our paper and the extensive experiments conducted.  

In the rebuttal below to each reviewer, we carefully address all the concerns raised. Additionally, we have provided a rebuttal file with figures and tables to enhance comprehension, specifically illustrating the inverse scaling law in relation to text tokens, model size, total training tokens against performance, and presenting the new evaluation results on VTAB benchmark.

Lastly, we are excited to share our latest scaling results, benchmarking our model against the current state-of-the-art. By training on DataComp-1B [1], our CLIPA-H/14 delivers an impressive performance of 81.8%, establishing itself as the best ImageNet-1k zero-shot model among released, open-source weights to date. Importantly, this milestone was reached with a training cost under $15,000 (measured using cloud A100 GPU price). This additional scaling result further confirms the efficiency and effectiveness of our proposed CLIPA. 

**References**

[1] Gadre, Samir Yitzhak, et al. "DataComp: In search of the next generation of multimodal datasets." arXiv preprint arXiv:2304.14108 (2023).


Meta-review: This paper studies the impact of token length (image and text) on the quality of image-language pretraining in CLIP-style models. The main finding and contribution of the paper is in CLIP-style training, the larger the model is, the shorter the token sequence is required to maintain a certain level of performance. Based on this finding, this paper proposes to train larger models with shorter sequences, which is more computationally favorable than training smaller models with sequences of full length.

This leads to 69.3% top-1 zero shot image accuracy with just eight A100 GPUs in ~4 days, which is 15x speed up over the conventional training protocols.

The paper shows many strong contributions as pointed out by multiple reviewers. 1) the work is well motivated. Reducing the cost of pretraining foundation models has a broad impact. 2) Given that CLIP is one of the founding breakthroughs of vision language models, the lessons identified in the paper may yield high impact. 3)  The writing is overall good. 4) Work provides a comprehensive analysis of various token reduction strategies for both image and text and also proposes new ones. 5) Analysis of different scales of models, across different masking strategies, is really helpful to convert a previously under-explored finding into a consistent phenomenon that can be useful for future research. 6) The work makes large model training accessible demonstrated by strong experimental results. Authors show that one can train the same architecture without much drop (e.g., 2-4% drop) but saving 10x-20x GPU hours.

Few weaknesses that is not fully addressed after the rebuttal is:
- The paper does not address the concern that the limited evaluation of the proposed model may hide model deficiencies compared to the original CLIP.
- Concerns on hyperparameter choice difference between proposed method and baselines
- Novelty vs "RECLIP"; (pointed out that this is concurrent so worth mentioning as a concurrent work with comparison) 

To elaborate, one of the main concerns raised from Reviewer `KzVG` is whether what the paper proposes is deemed to be called "inverse scaling law". Neither it is "quantitative" showing mathematical functional trend nor in conventional neural scaling law sense, as a function of optimal compute budget, size of model is increasing. "scaling law" is often reserved for the predictive formula which the paper lacks at the moment. 

While the term "scaling law" has not been very concretely defined in the field, the current form the author is using in the title and the paper is slightly misleading and confusing. AC strongly encourages this to be changed; i.e drop "law" (maybe replaced by "relationship" ) and be precise what is scaling with each other ( model size vs sequence length). The author used the term "inverse relationship between the model size and the length of image/text tokens in CLIP training" is more descriptive than "inverse scaling law". 

While the findings in the paper are interesting and practical, phrasing in terms of "inverse scaling laws" is not clearly representing the contribution and potentially misleading for people who are familiar with neural scaling laws. 

Another issue raised by `KzVG` and `UF9M` is whether evaluation is not sufficient to truly say reduced context training is neck-to-neck with the original method. This is quite important to characterize since the models trained with less compute budget may be missing something the full model learned. As CLIP-style model is used as pre-training to obtain interesting language-vision aligned representation, there could be strong downstream application impact. 

Final recommendations were split among the reviewers, three reviewers recommending acceptance (strong accept 8, accept 7, weak accept 6) and two reviewers recommending rejection (weak reject 4, reject 3).  While the concerns raised by two reviewers are valid, the strength of contribution and strong results outweigh potential concerns. As suggested above, AC strongly encourages the authors to rephase "inverse scaling law" as well as make suggested changes from the reviewers. On the topic of insufficient evaluation, AC believes study of better evaluation of representation learned through efficient training would make the paper strong and methodology more impactful but should not hinder publication.
Decision: Accept (poster)