Scaling (Down) CLIP: A Comprehensive Analysis of Data,Architecture, and Training Strategies
Authors: Zichao Li
Cihang Xie
Ekin Dogus Cubuk
Venue: Accepted by TMLR ()

Abstract: This paper investigates the performance of the Contrastive Language-Image Pre-training (CLIP) when scaled down to limited computation budgets. We explore CLIP along three dimensions: data, architecture, and training strategies. With regards to data, we demonstrate the significance of high-quality training data and show that a smaller dataset of high-quality data can outperform a larger dataset with lower quality. We also examine how model performance varies with different dataset sizes, suggesting that smaller ViT models are better suited for smaller datasets, while larger models perform better on larger datasets with fixed compute. Additionally, we provide guidance on when to choose a CNN-based architecture or a ViT-based architecture for CLIP training. We compare four CLIP training strategies - SLIP, FLIP, CLIP, and CLIP+Data Augmentation - and show that the choice of training strategy depends on the available compute resource. Our analysis reveals that CLIP+Data Augmentation can achieve comparable performance to CLIP using only half of the training data. This work provides practical insights into how to effectively train and deploy CLIP models, making them more accessible and affordable for practical use in various applications.

Published in Transactions on Machine Learning Research (04/2024)
Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategies
Zichao Li
2114890ucsc.co edu
Google Deepmind University of California, Santa Cruz
Cihang Xie
cirie edu
University of California, Santa Crus
Ekin Dogus Cubuk
cubukéigoogle.om com com
Google Deepmind
Reviewed on OpenReview: https: // openrevee........ net/ forum? id= t4nnCi5A06
Abstract
This paper investigates the performance of the Contrastive Language-Inage Pre-training (CLIP) when scaled down to limited computation budgets. We explore CLIP along three dimensions: data, architecture, and training strategics. With regards to data, we demon- strate the significance of high-quality training data and show that a smaller dataset of high-quality data can putperfrr larger dataset with lower quality. We also examine how model performance varies with different dataset sizes, suggesting that smaller VIT models are better suited for smaller datasets, while larger models perform better on larger datasets with fixed compute. Additionally, we provide guidance on when to choose a CNN-based architecture or a ViT-based architecture for CLIP training. We compare four CLIP training strategies SLIP, FLIP, CLIP, and CLiP+Data Augmentation and show that the choice of training strategy depends on the available compute resource. Our analysis reveals that achieve comparable performance to CLIP using only half of the training data. This work provides practical insights into how to effectively train and de- ploy CLIP models, making them more accessible and affordable for practical use in various applications.
CLIP+Data Augmentation
1 Introduction
In recent years, there has been surge of interest in image-and language representation learning (Radford et al. 2021 Jia al. 2021 Pham et al. 2021 Ramesh al.) 2022 which aims to capture the rich and complex interplay between visual and textual information. One of the most promising approaches in this area is the Contrastive Language- Image Pre-Tree Training (CLIP) framework, which leverages large-scale text and image data to learn a unified representation space for both modalities. CLIP has achieved state-of- stateoof-the-rt performance on a range of downstream tasks and has been shown to generalize well to out-of-distribution
data (Chen et al. 2019 Jia et al. 2021 Miller et al. 2021 Taori et al. 2020 Nguyen et al. 2022
Gontijo-
Lopes et al. 2021
While previous studics scaling Contrastive Language-Image Pre-Training (CLIP), such Li ct al. 2022
Cherti et al. (2022), have primarily focused large-computation scenarios, work aims to explore the performance of CLIP under resource constraints. We present a comprehensive study of scaling down CL.IP in three directions: data, architecture, and training strategies. Specifically, we investigate the effect of using different training data sizes, compare the performance of various architectures at different computation
* Work done during internship Google.

---

Published in Transactions on Machine Learning Research (04/2024)
budgets, and evaluate the effectiveness of different pre-training strategies. Our experiments conducted a large image-and-language dataset, WebLI (Chen et al. 2022). which contains over 3.4 billion image-text pairs in English. Particularly, we set a computation limit for most of our experiments, and the total number of sampled data for most experiments is at most 3.4B.
We train on datasets of different sizes to explore if the performance on ImageNet variants align with the performance on ImageNet. We also explore the importance of data quality and demonstrate that a smaller set of high-quality data outperform larger datasets. We find that using only the higher quality 40% of the data achieve better performance than using the entire dataset, and we investigate how model performance changes with increasing dataset sizes. Our results provide guidance on how to select training data for CLIP models, a critical issue for practical applications.
Concerning architecture, we compare multiple architectures under various dataset sizes. We investigate how choose size of the vision encoder depending on the dataset size and compute budget. We show that the a larger VIT (Dosovitskiy
al. 2020) is not always better. We also demonstrate the importance of selecting between CNN (He et al.) 2016) and VIT architectures for CLIP training. Although previous studies have shown that ViT-based CLIP has better performance than CNN-based CLIP models, we find that when the number of sampled data is small, CNNs perform better.
Finally, we compare four options: SLIP Muu et al. 2021 FLIP (Li et al. 2022 CLIP, and CLIP+Data Augmentation. We show that SLIP Mu et al. (2021) is not always the best choice compared to CLIP and FLIP. When the size of the training set is small, SLIP performs better than CLIP. However, as the training data size increases, SLIP has similar performance to CLIP but incurs twice the computational cost. Our rosults provide insights into the tradc-offs between computational cost and performance in CLIP training, critical issue for practical applications.
Finally, in terms of training strategies, we compare four options: SLIP Mu et al. 2021), FLIP Li et al. (0022) CLIP, and CLIP+Data Augmentation. We show that SLIP Mu et al. (2021) does not always outperform CLIP and FLIP. When the size of the training set is small, SLIP performs better than CLIP. However, at larger training set sizes, SLIP has similar performance CLIP, but requires twice the computational cost. We explore the tradc-offs between computational cost and performance.
100
WFIV1
No Data Aug Crop&Flip Rand/ug
WTS/16
50
VYFVV22
Error Rate
Accuracy
40
Stacked RA
60
WTL/11
Res50
30
Convinxt-7
20
40
SURWWE-RS2)
PLIPII 3032)
10
CatP+ Aug (VIBBB22
30
10a
10"h
10
10
10M
25M
50M
100M
Total Computation
Sampled data
(a) Total computation V.S. Zero-Shot ImageNet Error Rate
(b) Comparison between various data aug- mentation for CLIP
Figure 1: Training a High-Quality CLIP Model: This figure highlights the main contributions of our work. In Figure [[a] we demonstrate the relationship between different models, strategies, and error rates on the ImageNet dataset. The total computation is computed by GFT.OPs per sample times the number sampled data. Additionally, in Figure [(b] we illustrate how data augmentation methods improve the zero- shot performance of various datasets.
2 Related Work
Contrastive Language-Image Pre-Training: In recent years, the field of natural language processing (NLP) has made significant strides in pre-training language models on large amounts of text data (Peters
2

---

Published in Transactions on Machine Learning Research (04/2024)
et al. 2018
Devlin et al. 2019 Radford et al.
2018 Concurrently, computer vision has witnessed surge
in trainig convolutional neural net works CNNs) on large-scale image datasets (He et al. 2016 Huang Simonyan & Zisserman 2014). The CLIP model (Radford et al., 2021) is introduced as a utilizes a contrastive loss function to learn a shared embedding space between images and their corresponding lextual descriptions. CLIP has demonstrated remarkable zero-shot performance on various visual classification tasks, achieving accuracy levels comparable to those obtained by supervised learning methods.
et al., 2017
joint pre- pre-training framework for image and text representations.
Recently, LiT (Zhai et al. 2021 has been proposed as extension to CLIP. LiT introduces locked-imagee tuning, which adopts a pre-trained visual encoder and updates only the text model. As result, LiT can attain better zero-shot performances more efficiently. The success of CLIP has also inspired the work of SLIP
MMu et al. 2021). which combines supervupe learning with CLIP to enhance the visual encoder.
Scaling Language-Image Pre-training: In recent years, considcrable effort has been devoted to en- hancing the efficiency and scalability of the Contrastive language-Image Pre-training (CLIP) method. To improve scalability, [i et al. (2022) has proposed method called FLIP, which masks half or more patches of the training images to reduce computation by 2x and allow for the use of larger batch sizes, while maintaining the performance of the original CLIP models. In similar vein, Cherti et al. (2022) has provided further insights into the scaling laws of CLIP models, revealing that the training distribution plays a pivotal role in scaling laws, as evidenced by the different scaling behaviors exhibited by OpenCLIP [lharco et al. (2021) models, despite having identical model architectures and similar training procedures.
Research on scaling CLIP models has predominantly focused on training with large computational resources, often starting from ViT large model and training massive datasets over multiple epochs. For example, the largest number of sampled data in Cherti et al. (2022) is 34 billion. Less attention has been paid to the performance of CLIP models with smaller training budgets.
3 Methods
Training Pipcline We adopt the same training pipcline CLIP (Radford et al., 2021 which employs contrastive loss to Irain both the vision encoder and text encoder from scratch. The contrastive loss encour- ages the vision and text encoders to map corresponding image-text pairs to similar feature representations in a shared embedding space. Specifically, the contrastive loss minimizes the distance between the represen- tations of positive pairs (i.e., image-text pairs that belong together) while maximizing the distance between the representations of negative pairs (i.e., image-text pairs that do not belong together).
Dataset and Data Pre-processing We conduct our experiments on WebLl Chen et al. (2022), a large image-and-language dataset built from the public web, containing over 10 billion image-text pairs from more than 100 languages. We specifically focus on pairs in the English language, resulting in a dataset of approximately 3.4 billion pairs. In our base experiments (referred to as just CLIP), we do not apply any data augmentation to images, except for resizing it to 224x224 and normalizing pixel values to the range of -1 to!! 1.
For text processing, we use a SentencePiece tokenizer with a vocabulary size of 32k. We set the token length to 16 and pad or truncate sentences to that length.
3.1 Hyper-parameters
In our study, we adopt the hyper-parameler settings used in a previous work Zhai el al. (2021 We use AdafactofShazeer & Stern (2018) as the optimizer with 31 and 82 set to 0.9 and 0.999, respectively, and set the batch size of all our models to 16k. To adjust the learning rate, we use a cosine learning schoduler with an initial rate of 0.001, and we set the weight decay to 0.0001. These hyper- paramtpar are carefully selected to ensure optimal performance of the CLIP model in our experiments.
3

---

Published in Transactions on Machine Learning Research (04/2024)
3.2 Evaluation Metrics:
In the zero-shot transfer evaluation, we aim to assess the model's ability to generalize to new tasks without any fine-tuning. To ensure a fair comparison with prior work, we follow the same evaluation settings as Radford et al. (2021); Zhai et al. (2021). We use the same prompts collected from these works and pre- process the test images by first resizing them and then applying central crop with 0.875 aspect ratio to match the target resolution.
For the linear probe ovaluations, we freeze the vision encoder and zoro-initialize the fully connected layer. We then sweep the learning rale to find the best rate for each architectures.
For the evaluation of retrieval performance on MSCOCO captions, we report the results based on the test set. For image-to-text retrieval metrics, we rank all text captions in descending order basod on the cosine similarity of their embeddings with the image embedding. Subsequently, we report the proportion of images that achieved the Recall@1 performance. For text-to-image retrieval, we follow similar approach but rank the images instead and calculate the average results across all texts.
4 Data
Data Quantity To assess the influence of data quantity on performance, we conduct. experiments using five datasets of varying sizes: 10M, 25M, 100M, 200M, and 400M. We choose the 400M as the largest dataset in this section as it is commonly used in CLIP training. For all CLIP models in this scction, we use ViT-B/32 as the vision encoder. We train models for number of epochs ranging from 2 to 32.
The findings regarding the zero-shot performance of CLIP models on ImageNet are depicted in Figure ( The experiments reveal that the impact of training data quantity model performance is contingent upon both the dataset size and the number of training epochs. In particular, we note that for smaller datasets such as the 25M dataset, increasing the training epochs does not yield significant. performance improvements on ImageNet. However, for larger datasets like the 400M dataset, more training epochs do result in enhanced performance on ImageNet.
Additionally, we evaluate the average zero-shot performance of our models on ImageNet variants, illus- trated in [ggure[(b) The average zero-shot performance across six ImageNet variants follows similar trend to that observed on ImageNet: larger datasets and longer training result in improved performances. How- ever, we observe that the performance trends on ImageNet variants do not consistently correspond to those on ImageNet for cach specific variant, as shown in Figure 20 in Appendix. For instance, while increasing the number of epochs from 16 to 32 does not improve performance on ImageNet for the 25M dataset, it leads to improved performance on TmageNet-A. Similarly, our models trained for 32 epochs exhibit significantly better performance on ImageNet-R and ImageNet-Sketch compared to those trained for 16 epochs, despite improvement on ImageNet.
Additionally, we conduct experiments on few-shot and retrieval scenarios, and the results pertaining to few- shot and retrieval performances across different dataset sizes are depicted in Figures 3 and 4 We observe that the few-shot performances demonstrate a similar trend to the zero-shot performances on ImageNet. However, we notice a slightly distinct trend retrieval performances. Specifically, we note that when the number of epochs surpasses cight, there minimal improvement in both image retricval and text retrieval performances.
Data Quality We compute the similarity between images and texts to create subsets from the 3.4B dataset, containing the top 20%, 40%, 60%, and 80% of the highest quality data. These filtered datascts are then used to Irain models for a single epoch and evaluate their zero-shot performance on ImageNet. The findings from our study, as depicted in Figure 5[a] suggest that the number of samples encountered by the CLIP model does not always have the greatest influence on performance. Interestingly, despite significantly fewer iterations than the full dataset, the model trained on the Top40% subset demonstrates notably superior performance.
4

---

Published in Transactions on Machine Learning Research (04/2024)
SIN
90
585
Error Rate
80
1839
70
80
1122
70
60
40
16
Epochs
Epochs
(a) Training Epochs V.S. Ima-
(b) Training Epochs V.S. Av-
geNet Accuracy
crage
ImageNet Variants Error
Rate
Figure 2: Data Quantity: Zero-Shot performances with the same dataset size across varied training epochs
95
othot Error Rate
10Sbot Error Rate
85
80
75
70
65
60
1000
55
50
Epochs
Epochs
(a) 5 Shot
(b) Shot
Figure 3: Data Quantity: Few-Shot Porformances on ImageNet.
To ensure a fair comparison among datasets of varying quality, we also train datasets with an equal number of sampled data points, and the results are presented in (igure5((b) Our findings suggest that, on ImageNet, the Top40% dataset achieves the highest performance, whereas the Top20% dataset falls short of the Top40% in terms of performance. These results emphasize the importance of data quality in training effective CLIP models and highlight the potential benefits of focusing on high-quality data subsets in this context.
We display the few-shot and retrieval performances on datasets with different qualities in Figure and Figure 7 respectively. The results indicate that datasets with higher quality yield superior few-shot and retrieval performances, even with relatively small number of sampled data points. Moreover, when the number of sampled data points consistent, datasets of higher quality demonstrate superior 5-shot and 10- shot performances. Particularly, the top 80% dataset exhibits the most impressive retrieval performances.
5 Variants of Vision Transformers
In this section, aim is to investigate how the performance of different CLIP models, which vary in the size of their vision encoders, is affected by dataset size and the number of sampled data. To achieve this goal, we present the zero-shot performance and linear probing results of these models, providing a more comprehensive understanding of the impact of dataset size and the number of data samples on CLIP's performance.
To investigate the effect vision encoders on CLIP's performance, we fixed the size of the text transformers vit-base and experimented with various vision encoders, including VIT-Ti/16, S/16, B/32, B/16, and L/16 (Dosovitskiy al. 2020
The literature on scaling CLIP typically demonstrates power law starting with a large computation cost. To evaluate CLIP's performance as we scale up the datasel size, we sampled ten subsets from the entire dataset, with sizes ranging from 10M to 3.4B. These subsets have the same data distribution and quality and are used in this scction and subsoquent sections. To assess the effect of the number of sampled data, we provide results for all subsets trained for one epoch. To ensure a fair comparison, we also provide results for all subsets trained for the same number of iterations, resulting in the same number of sampled data.
5

---

Published in Transactions on Machine Learning Research (04/2024)
1mg2Txed 100 Remalle
96
80
66
70
25
3055
60
16
size
Dataset
(a) Text Retrieval
(b) Image Retrieval
Figure 4: Data Quantity: Retrieval Performances on MSCOCO Chen al. 2015
Ervor Rate
38
40
37
39
36
35
40 70 90
60
100
Top Quality Data Threshold (%)
Top Quality Data Threshold (%)
(b)
Figure 5: Data Quality: Zero-Shot Performances on ImageNet. Figure 5(a) shows results trained for one epoch. Figure ((b) shows results trained for the same number of sampled data. We use ViT-B/32 the vision encoder and ViT-B as the text encoder.
Zero-Shot Figure S(a) showcases the zero-shot performances on ImageNet after training for one epoch using various sizes of training sets with different ViT models. The results indicate that ViT-L/16 does not consistently outperform other ViT models. In fact, when the number of samples is less than 100 million, ViT-L performs the worst among the ViT family. However, the size of sampled data increases, larger vision encoders exhibit better performances. We conduct experiments number of iterations, keeping the number of sampled data constant. Our results reveal that a larger dataset yields better performance, even when the number of iterations is held constant. Notably, we observe that augmenting the dataset to 400M does not yield any significant improvement in zero-shot performance due our computation limits. Moreover, we discover that the performance gap between larger ViTs and their smaller counterparts widens the dataset size increases.
different dalasets for equal
Our experiments also investigate the relationship between zero-shot performance ImageNet and other distributional datasets. We find that the accuracy curves for ImageNet-R (Hendryeks et al. 2020). ImageNet- Sketch (Wang et al. 2019), ImageNet-V2 (Recht et al. 2019). and ObjectNet (Barbu et al. 2019) are nearly linear as shown in Figure 21 in Appendix, indicating that the image classification ability of CLIP models improves consistently across these datasets. However, the accuracy of ImageNet-A (Hendrycks al., 2019
does not exhibit a lincar relationship with that of ImageNet (Deng et al.) 2000) as shown Figure (1 in Appendix. Specifically, when ImageNet accuracy is below 50%, the accuracy of ImageNet-A increases only slightly, while it grows more rapidly for higher ImageNet accuracies. We attribute this behavior to the presence of more challenging test examples in ImageNet-A, which is not present in the other four datascts that only contain variants of ImageNet images. We also find that for most of OOD datasets, when the standard accuracy is same, a larger ViT will have better robustness performances.
Linear Probing As shown in Figure 10 we present the linear-probing results and obscrve that for smaller datasets, the performance of ViT-L/16 is inferior that of the other three models. However, for larger numbers of sampled data, the larger VIT models deliver significantly better results.

---

Published in Transactions on Machine Learning Research (04/2024)
50
Error Rate
Rate
Ecrur Rate
12
20 30 40 50 60 70 80 100
20 35 40 50 60 70 80 105
20 30 40 50 60
130
25 30 40 50 do 75 20 100 Top-Quality Data Thmshold (%0
Top-Quality Data Threshold (8)
Top-Quality Data Threshold (%)
Top-Quality Data Threshold (96)
Shot-one epoch
(b) 10 Shot-one epoch
shot-same itera-
(d) shat-same itera- tions
tions
Figure 6: Data Quality: Few-Shot Performances on ImageNet. We train dataset for various quality thresholds for one epoch in (a) and (b). We train dataset for various quality thresholds for the same number of sampled data (c) and (d).
"xtlmmtt wo..ottooBcC.. Recallis
Imgg 100 DO Bocallet
02
58
%
66
20
20 30 40 30
70 80 30 100
20 30 40 50 105
Top-Quality Data Thresholl (9)
Top-Quality Data Threshold (%)
Quallyylty Data Thresbold (%)
Top Quality Data Threshold (%)
(a)
Text
Retrieval-one
(b) Image Retrieval-one (c) Text Retrioval-same
(d) Image Rotricval-samee
epoch
epoch
iterations
iterations
Figure 7: Data Quality: Retrieval Performances on MSCOCO. We train dataset for various quality thresh- olds for one epoch in (a) and (b). We train dataset for various quality thresholds for the same number of sampled data in (c) and (d)
Furthermore, in Figure 22 in Appendix, compare ImageNet with other out-of-distribution (OOD) variants and find that the performance of vision encoders significantly declines for most OOD variants, except for ImageNet-V2.
Our results also show that for smaller ViTs, the effect of data diversity is smaller. For Ti/16, when the dataset size is larger than 25M, the increase dataset size will not benefit the linear probing performance. However, for L/16, the increase of data diversity will benefit the lincar probing performance until the training dataset size is larger than 400M. These findings demonstrate the importance of using large datasets for larger CLIP models.
Retrieval The results shown in Figure 11 demonstrate that ViT-L/16 exhibits poor performance for retrieval tasks when the number of sampled data points is less than 100M. However, with an increase in the number of sampled data points, larger ViT models demonstrate improved performance.
Furthermore, when the number of sampled data points is held constant, the trend retrieval performances aligns with that of zero-shot performance. Moreover, as the dataset size increases, the performance of smaller ViT models plateaus earlier than that of larger models, suggesting that larger models derive greater benefits from additional data.
6 A Comparison of Network Architectures
Previous studies Zhai et al. (2021 Radford et al. (2021) have investigated different vision encoders for CLIP.
including ResNet He et al. (2016). MLP-Mixer Tolstikhin el al. (021)), and ViT Dosovitskiy el al. (2020) However, architectures such as Swin- Transformer Liu et al. (2021) and ConvNext Liu et al. (2022) have not been explored. Moreover, previous works have only compared network performance for only one training setting. In this section, we compare several CNN and vision transformer architectures with similar FLOPs, namely ViT-B/32, ResNet-50, ConvNext-T, Swin-T, and Mixer-B/32.

---

Published in Transactions on Machine Learning Research (04/2024)
70
RV16
100
1055
Accuracy
205
LAS
60
40
10
30
25M
50M 00M 1B
3.48
10P
10"
10
10
Sampled data
Total Computation
(a) Sampled data V.S. ImageNet Accuracy
(b) Total Computation V.S. Im-
ageNet Error Rate
Zero-Shot performances with various numbers of sample data. We use various vision encoders and the same text encoder ViT-B. The total computation is computed by GFLOPs per sample times the number sampled data. The total computation computed by GFT.Pss per sample times the number sampled data.
Figure 8: Various ViTs:
100
100
Eror Rate
60
40
LO6
107
10a
10
30 107
10a
10
Dataset size
Dataset size
(a) Data size V.S. ImageNet Er-
(b) Data size V.S. Average Tma-
ror Rate
geNet Variants Error Rate
Figure 9: Various ViTs: Zero-Shot performances with the same number of sampled data: 3.4B. We use various vision encoders and the same text encoder ViT-B.
Zero-Shot We begin our investigation by examining the performance of different architectures for different numbers of sampled data. As illustrated in Figure 12(a) we observe that ResNet-50 outperforms ViT-B/32 when the number of sampled data is limited. However, the number of samples increases, ViT-B/32 achieves better performance. One intuitive explanation is that CNNs possess stronger inductive bias for images and capture more local information even with small number of samples. Moreover, find that the performance of vision transformers variants does not show notable improvements when trained on a limited number of samples, and in most cases, Swin-T's performance is inferior to that of VTT-B/32. We
plot the total computation cost versus error rate in Figure 12(b) When the total computation cost is low, Re outperforms other architectures.
90
90
Error Rate
B02
20
40
Error
60
50
20
10F
10a
10'
10a
107
Dataset Size
Dataset
(a) Data size V.S. ImageNet
(b) Data size V.S. Average on
OOD datasets
Figure 10: Various ViTs: Linear probing performances with varions sizes of vision encoders. We train all subsets with the same number of sampled data: 3.4B.

---

Published in Transactions on Machine Learning Research (04/2024)
Imggerennon Rocalle
100
4.0
Tx 20020aCC00---raalll
50
30
70
60
100M0O0M
100M-00M 18
16
Sampled data
Sampled data
Dataset size
Dataset Size
(a)
Text
Retrieval-one
(b) Image
Retrieval-one
(c) Text Retricval
(d) Image Retrieval
epoch
epoch
Figure 11: Various VITs:
Retrieval Performances MSCOCO.
Next, we evaluate the effect of dataset size on the zero-shot performance of different architectures, while keeping the number of iterations constant. As demonstrated in Figure 13 ViT-B/32 consistently delivers the best performance. We observe that the impact of data diversity is similar across different networks, and increasing the dataset size beyond 400M does not offer any significant benefit to the zero-shot performance of any network. Therefore, the improvement in zero-shot performance due to data diversity is mainly related to model size rather than model architecture.
We also present the zero-shot performances of ImageNet and ils variants in Figure 23 When the Ima- geNet accuracy is comparable, Swin-T and ViT-B/32 exhibit better performances on ImageNet variants, particularly on ImageNet-A. Our findings suggest that when the training dataset is sufficiently large, vision transformers provide better choice the vision encoder of CLIP.
100
Intoc
Accuracy
40
MLL Nav
30
Error
60
20
10
Sampled data
Total Computation
(a) Sampled data V.S. ImageNet Accuracy
(b) Total
Computation
V.S. V.S.meee Error
Rate
Figure 12: Various architectures: Zero-Shot performances with various numbers of sampled data.
We train them for one epoch.
Andn
Satoc
Error Rate
80
70
50
10'
10
10*
Datasel
Dalasel
Data size V.S. ImageNet Er-
(b) Data size V.S. Average Ima-
Rate
geNet Variants Error Rate
Figure 13: Various architectures: Zero-Shot performances with various dataset sizes and the same number of steps.
Linear Probing To validate the performance various encoders in CLIP models, we evaluate them using lincar probing and present the results Figure It can be observed that when the number of sampled data is below 100 million, MLP-Mixer outperforms the other four architectures on both ImageNet and other OOD variants. However, the number of sampled data increases, ViT-B/32 exhibits better results, particularly

---

Published in Transactions on Machine Learning Research (04/2024)
OOD variants. These results provide insights for selecting appropriate feature extractors for datasets with different sizes. Specifically, ResNet is a better choice when the number of sampled data is small, while ViT can achieve better robustness and standard accuracy when the number of sampled data is sufficiently large.
In Figure Appendix, we present the results of linear probing on ImageNet and various OOD datasets. Notably, we observed a significant drop in performance on ImageNet-A,, -R, and ObjectNet after linear probing, whereas the performances is close, noticed that VIT and MLP-Mixer demonstrate better robustness. We hypothesize that this might be due to the relatively lower inductive bias of these two architectures, meaning they are less prone to overfitting to the specific dataset they were trained on. This can lead to better generalization and performance on oittof-distribution datasets.
ImageNet-V2 and -Real improved. When the standard accuracy
100
100
90
Error Rato
70
60
30
107
10a
10
10a (b) Data size V.S. Average
50 10
10a
Sampled data
Sampled data
(a) Data size V.S. ImageNet
OOD datasets
Figure 14: Various architecturcs:
Lincar Probing with the same number of sampled data.
Retrieval When considering different numbers of sampled data points, ResNet-50 demonstrates superior performance for smaller sample sizes, as depicted in Figures 15(a) and 15(b) However, as the number of sampled data points exceeds 400M, VIT-B/32 surpasses ResNet-50 in performance for both few-shot and retrieval tasks.
Moreover, when the number of sampled data points is consistent, Mixer-B/33 exhibits the poorest per- formance for retricval tasks, shown in Figures 15(c) and [5(d) These results indicate that VIT is the preferred choice for the vision encoder not only in zero-shot and linear probing tasks but also in few-shot and retrieval tasks.
Fadd
1133901 100 Recalig1)
Tea22lngtt Loc (o--Becalllgg
94)
Img2rvt
80
80
20
75
m
60
60
50X JOONSOOM
5.45
15M
50X
1000400M
18
248
10'
10
10P
107 101
16
Sampled data
Sampled
Datisee
Dataset
(a) Text
Retrieval-one
(b) Image Retrieval-one (c) Text Retrieval-same iter.
(d) Image Retrieval-same
epoch
epoch
iter.
Figure 15: Various Architectures: Retrieval Performances on MSCOCO.
7
Training Strategies
In this section, we compare the training strategies of SLIP Muu et al. 2021 and FLIP Li et al. 2022 with the original CLIP. We use ViT-B/32 as the vision encoder and VIT B as the text encoder for all three strategics. We follow the implementation of SLIP as described in Mu ct al. (2021). For FLIP (Li et al. 2022), we apply 50% patch masking and double the batch size. We also propose our own training strategy, CLIP+ Data Augmenlation.
10

---

Published in Transactions on Machine Learning Research (04/2024)
CLIP + Data Augmentation While Mu et al (2021 proposed enhancing the vision encoder of CLIP models through self-supervised learning, this comess at significant computation cost compared to the original CLIP. To achieve similar enhancements to the vision encoder while saving on computation, we propose applying data augmentation to input images of CLIP. We validate the effectiveness of this approach on Tour subsets, training them for 30 epochs to ensure model convergence. We apply three data augmentation methods: crop&flip, RandAugment (Cubuk al., 2019), and Stacked RandAugment (Khosla al., 2020). The results, shown in Figure [(b)] demonstrate that all three methods improve performance compared to raw CLIP, highlighting the effectiveness of data augmentation in CLIP. Moreover, data augmentation input images does not bring any extra computation cost compared to CLIP. Even when trained a dataset of 25 million samples, the Stacked RA model can achieve comparable performance to that obtained with a dataset of 50 million samples.
SLPP
100
Accuracy
30
20
SLIT
40
25M
50M
M00M400 18
34B
30
107
10'
10
Sampled data
Total Computation
(a) Sampled data V.S. ImageNet Accuracy
(b) Total Computation V.S. Im-
ageNet Error Rate
Figure 16: Various Training Stategies: Zero-Shot performances with various numbers of sampled data. We all subsets for one epoch. We use ViT-B/32 as the vision encoder and ViT-B as the text encoder for strategies.
100
Error Rate
00
60
70
40
10"
10"
10
50
10
10"
10a
Dalasel size
Dalasel size
(a) Data size V.S. ImageNet
(b) Data size V.S. Average on ImageNet Variants
Figure 17: Training Stategies: Zero-Shot performances with various subsets while the same number of
sampled data. We use ViT-B /22 as the vision encoder and ViT-B as the text encoder for all strategics.
50
Imp2ime1o0 Rocalig3:
Foogl Ta 10-Ruccllige
80
30
88
60
70
10
5
63
0557
50M
100MUOUM 15 3.40
2552
50M
302MADOM 115
3.45
50
50y5"
16
10P
161
18
Sampled data
Sampled data
Datinet size
Dalasel size
(a) Text
Retrieval-one
(b) Image epoch
Retrieval-one
(c) Text Retrieval-same
(d) Image Retrieval-same
epoch
iterations
iterations
Figure 18: Training Strategies: Retrieval Performances MSCOCO.
Zero-Shot Our experimental results on the ImageNet dataset, as depicted in Figure 16(a) demonstrate that SLIP outperforms the other two strategies when the number of training samples seen by models is less than billion. This suggests that self-supervised learning for the vision encoder be beneficial when the
11

---

Published in Transactions on Machine Learning Research (04/2024)
90
90
CLI CLIP-DaY AAD
80
CUT
Error Rate
60
70
Error
60
107
10
10
50
10"
10
Dataset Size
Dataset
(a) Data size V.S. ImageNet
(b) Data size V.S. Average on
OOD datasets
Various Training Stategies: Linear probing performances with various subsets while the same number of sampled data.
Figure 19:
number of training samples is limited. However, as the number of training samples increases, both CLIP and FLIP achieve similar results and outperform SLIP. These results indicate that enhancing the vision encoders may not be necessary when the training dataset is large. Notably, when the model architecture is the same, SLIP is two times more computationally expensive than CLIP. As illustrated in Figure [6(b] when the total computation cost. is the same, SLIP has the worst zero-shot performance on ImageNet. We also observed that data augmentation hurts the zero-shot performance when we only train these subsets for one epoch.
Moreover, we investigated how the four training strategies perform with various dataset sizes, with fixed number of sampled data. As shown in Figure 17 SLIP outperforms CLIP and FLIP when the dataset size is small ImageNet. However, the dataset size increases, SLIP does not show better performance ImageNet variants. On the other hand, CLIP + Data Aug performs better than the other three training methods, especially on ImageNet variants. Our results suggest that data augmentation on input images can bring better generalization performance with almost no extra cost.
In Figure 26 in Appendix, we present the results of our study on ImageNet compared to various OOD variants. SLIP also does not show better performance on these variants when the ImageNet accuracy the same. However, CLIP Data Aug brings better performance ImageNet-Sketch and -R. In conclusion, can say that CLIP Data Aug can bring better zero-shot performance on both ImageNet and its variants when we train multiple epochs for the dataset.
Linear Probing We evaluated vision encoders trained with various strategics using lincar probing and found that those trained with CLIP + Data Aug consistently outperform the other three strategies across all data points, particularly on OOD datasets as shown in Figure 19 The performances of ImageNet and OOD variants are compared in Figure [7 in Appendix, where CLIP and CLIP Data Aug show better robustness than SLIP when ImageNet accuracy is the same. Considering the linear probing performances and computation cost, we believe that combining CL.IP and data augmentation can result. in a better feature extractor.
Retrieval The performance of SLIP, CLIP+Data Aug, CLIP, and FLIP evaluated for few-shot and retrieval tasks on ImageNet with varying numbers of sampled data. Moreover, SLIP consistently outperforms the other mothods on both image and text retrieval tasks as shown in Figures 18(a) and 18(b)
When the number of sampled data is fixed, the performance of each method for few-shot and retrieval tasks shows a different trend compared to the zero-shot performance. For retrieval tasks, SLIP consistently performs best across all datasct sizes as shown in Figures 18(c) and [8(d) These results suggest that SLIP is a better strategy for retrieval (asks, although it may not. be the best one for classification lasks.
8 Conclusion
In this paper, we investigate the impact of data size, network architecture, and training strategics on the performance of CLIP. Our experiments demonstrate the importance of data quantity and data quality. We also show that data augmentation techniques enhance the performance of CLIP without significantly
12

---

Published in Transactions on Machine Learning Research (04/2024)
increasing the computational cost. Additionally, explore the impact of different network architectures and training strategies the performance of CLIP. Our results suggest that some architectures and training strategies perform better than others at different computational budgets, highlighting the importance of careful selection.
9
Acknowledgements
We acknowledge Emirhan Kurtulus for helpful discussions. We also acknowledge Douglas Eck, Andrew Pierson, and the rest of the Google DeepMind team for their support.
References
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Joshua B. Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Neural Information Processing Systems, 2019.
Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel M. Sal,, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lueas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language- image model. ArXiv, abs/2209.0625 02794, 2022.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/ abs/1554.000325, 2015.
Cen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference Computer Vision, 2019.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhrnann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. ArXio, abs/ abs/2212.071443, 2022.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with a reduced search space. 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1008-3017, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.
Raphael Gontijo-Lopes, Yann Dauphin, and Ekin D Cubuk. No one representation to rule them all: Over- lapping features of training methods. aix preprind arXiv:21 10. 12899, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770- 770-778, 2016.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. 2021 IEER/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 15257- 15266, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustuess: A critical analysis of aut-of-distribution generalization. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 8320 8320-8329, 2020.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Gabriel Mharco,, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Lud- wig Schmidt,. Openclip; July 2021. URL https://doi. http:/dii.ogg////// www.oo.g.......... /tt.//00.52///eooo55554447/ If you use this software, please cite it as below.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun HanaH Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. ArXie, ////20004.13622 2020.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pro-training via masking. ArXiv, abs/22122 Abss2212.00094, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han HU, Yixuan Wei, and Zheng Zhang. Swin transformer: Hicrarchical vision transformer using shifted windows. aixio preprint arXiv:2103. 14030, 2021.
Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11966 11976, 2022.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between dutrof-distribution and in-distribution generalization. ArXiv, abs//2107006499 2021.
Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. Slip: Selessipeiiiionn meets language- image pre-training. ArXiv, abs/2112 abs/2112.12750, 2021.
Thao Truong Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. ArXie, hbss/220.005516, 2022.
Matthew F. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zeettlemoyer. Deep contextualized word representations. arXiv preprint 1rX2 05365, 2018.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoe V. Le. Combined scaling for zero-shot transfer learning. ArXiv, abs/2111.10050, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training. OpenAl Blog, 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πya Sutskover. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400, 2019.
Noam M. Shazeer and Mitchell Stern. Adafactor: Adaptíve learning rates with sublinear memory cost.
ArXio, abs/ 100015,,,,4 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arxii:14090 1556, 2014.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt, Mea-
suring robustness to natural distribution shifts in image classification. ArXiv, abbs/2007.0044 2020.
llya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In Neural Information Processing Systems, 2021.
Haohan Wang, Songwei Ge, Eric Xing, and Zachary Chase Lipton. Learning robust global representations by penalizing local predictive power. In Neural Information Processing Systems, 2019.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. 2022 IEEE/CVP Conference on Computer Vision and Pallern Recognition (CVPR), pp. 18102 18112, 2021.
15


Reviews:
[{"review_id": "h5yZVc1kex", "reviewer": "Reviewer_tNM9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The findings from the paper provides a comprehensive guideline on how to train CLIP models considering a broad range of factors affecting model performance.\n\nThe following can be both the advantage and disadvantage of this work:\n- Even though the authors set the maximum number of samples, conducting all the experiments reported in the paper requires a huge amount of training costs that only a very few labs can replicate them. Unless the authors release all training checkpoints, it is very hard to validate the experimental results.\n\nWeaknesses: overall, writing should be improved. There are incorrect or missing details mentioned in the paper.\n- It seems that the paragraphs after \"containing the top 20\" and \"the Top40\" are commented out in Section 4.\n- The original CLIP uses a relatively small text encoder while this work uses the same size as the vision encoder with very shot token length (16). Is there any reason for this?\n- Please add the details of each of six ImageNet variants.\n- In Section 4, the authors said that longer training epochs led to enhanced performance on ImageNet only with large datasets. On the contrary, however, the performance gain on longer training decreases as the dataset size increases according to Figure 2.\n- In Section 5, the authors said that they sampled ten subsets of varying sizes from WebLI, and all the subsets had the same data distribution and quality. Please add the details about how to ensure the same data distribution and quality.\n- The authors used the term \"diversity\" in Section 5 and Section 6: they said the diversity increased when the number of sampled data points increased. However, this is not true because all the subsets have the same data distribution.\n- In Section 6, the authors said that ResNet-50 outperformed the other architectures on linear probing when the number of sampled data is below 100 million, but this is not true: Mixer-B/32 is the best according to Figure 14.\n- The authors mentioned the few-shot classification task when comparing different model architectures and training strategies, but the experimental results on the task are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper extensively explores CLIP training along three dimensions: training data, model architecture and training strategy, and provide several findings especially useful for training CLIP models with a limited computation budget. The most notable findings from experiments on zero-shot transfer, linear probing, few-shot classification and image-text retrieval are as follows:\n\n- Dataset quality is quite important: training on a smaller dataset of high quality outperforms training on a larger dataset of lower quality.\n- We should increase the dataset size for fully benefitting from increasing the ViT model size.\n- CNN models perform better with a smaller dataset due to its high inductive bias while ViTs perform better with a larger dataset.\n- While SLIP performs better when training on a smaller number of samples compared to FLIP and CLIP (+ augmentation), CLIP with strong data augmentation performs best when training on a larger dataset, except in the task of retrieval: SLIP performs best across all dataset sizes on image-text retrieval.\n\nThese findings will provide guidance on how to train CLIP models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please modify the draft to reflect what I mentioned above. Also, it would be very good to compare different architectures, training strategies and model sizes when training on datasets of different distribution: e.g., sampling the same number of data from WebLI and LAION-5B [1].\n\nMinor request: the paper addresses training CLIP models on a wide range of data scales, even beyond the original dataset scale (400M). Thus, I recommend changing the title of the paper: it is too narrow compared to the broader scope the paper actually addresses.\n\n[1] Schuhmann et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In NeurIPS Datasets and Benchmarks Track, 2022.", "broader_impact_concerns": "There are no specific ethical issues regarding the paper, but please add the broader impact of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nk2X9uMgBX", "reviewer": "Reviewer_qMQC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- Extensive analysis of various training configurations for CLIP.\n- Enormous computational resources are utilized for this training.\n\n\n## Weakness\n\n- Additional relevant works\n    - This paper reminds me of the analysis papers like [1,2]. It would be great to discuss their findings and clarify the differences from this work. For example, [1] studies ViT in the vision-only setup, while this paper focuses on vision-language training. If different trends emerge between vision-only and vision-language setups, it would be worth highlighting them.\n    - A minor point, but there are more works on scaling down CLIP [3].\n    - Another minor point, but the effect of data quality is studied in recent data curation works [4,5].\n\n[1] Steiner et al. \"How to Train Your ViT? Data, Augmentation, and Regularization in Vision Transformers.\" TMLR 2022.\\\n[2] Hendricks et al. \"Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers.\" TACL 2021.\\\n[3] Li et al. \"An Inverse Scaling Law for CLIP Training.\" NeurIPS 2023.\\\n[4] Xu et al. \"CiT: Curation in Training for Effective Vision-Language Data.\" arXiv 2023.\\\n[5] Xu et al. \"Demystifying CLIP Data.\" arXiv 2023.\n\n- Observations are not surprising\n    - It is intuitive that smaller ViTs and CNNs are better for smaller datasets. Also, SLIP and CLIP + data augmentation would improve CLIP but can lead to slower convergence. Nevertheless, I think the analyses are worth reporting, especially the comparisons with CLIP, SLIP, and FLIP. It would be more interesting to see other training strategies like CoCa and BLIP though.\n    - As mentioned above, if the paper highlights the findings that are specific to vision-language, it would be more exciting.\n\n- Considerations on data augmentation for CLIP\n    - As far as I know, a tricky aspect of applying data augmentation to CLIP is that the augmentation may disrupt the alignment of images and text. For example, RandomCrop may remove some objects from the caption, which still remains in the caption. Can the authors comment on this?\n\n- Editorial comments\n    - Check the \"data quality\" part in Section 4. It seems that the paragraphs after \"%\" are commented out. Also, please maintain consistent usage of \"top 20\" vs. \"Top 40\".\n    - Some \\citep references are not spaced from the previous words. Check pages 5 and 6, for example.\n    - Figures are too small. Maybe the authors can consider increasing the figure size and placing less important ones in the Appendix.\n    - TMLR can merge the Appendix into a single PDF, which would enhance readability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines various training configurations for CLIP. Specifically, the paper reveals that 1) small high-quality data is better than large low-quality data, 2) small ViTs perform better than large ones with smaller datasets, 3) CNNs are better than ViTs in small-scale regimes, 4) SLIP and data augmentation improve CLIP but lead to slower convergence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Emphasize the findings specific to vision-language.\n- Address the editorial comments.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Oo9l6QLcis", "reviewer": "Reviewer_MqEg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* Besides some issues where text appears to be missing/incomplete, this work is relatively clear and provides some intriguing insights into the behaviour of CLIP, and the impact of a wide variety of design choices in certain resource capped settings.\n* This work will be of interest to a wide audience given the broad applicability and use of CLIP features in the literature.\n* This work complements existing scaling results on CLIP by focusing on settings where the number of training iterations, model sizes, data size, etc. is small. It is akin to an inverse scaling ablation of CLIP.\n\n**Weaknesses**\n\nI provide detailed suggestions/comments on the manuscript in the other sections below, but the major weakness in my opinion is the lack of a coherent story or messaging to the work. While this work covers a broad range of ablations, it does not dive deeper into any of them. Rather, it feels as if there are many interesting one-off experiments that have been stitched together into a technical report, and noticeably lacking is an in-depth study of any one of these ablated factors.\n\nFor instance, Figure 6 (c)/(d) regarding data quality is particularly interesting, but why does similarity thresholding on the image-caption pairs lead to better 5-shot or 10-shot IN1k performance? Are there other ways of measuring data quality? What training schedule is used in Figure 6 (c)/(d) and how would the findings change with different schedules (longer vs shorter)?\n\nAnother example would be diving deeper into the model architecture experiments. For instance, being able to provide measurable experiments that explain why a RN50 would outperform a ViT-B/32 with limited data and training iterations would be helpful beyond simply recording this observation.\n\nIn short, it would be more helpful to dive deeper into any of these experimental settings and formulate and test hypotheses about the expected behaviour of models in such settings, rather than simply recording results in various settings. In my opinion, a depth rather than breadth approach would be particularly helpful for improving the impact of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides an in-depth exploration of CLIP (and variants, such as FLIP and SLIP) in a variety of settings, varying the model architecture, training iterations, and number of training samples filtered according to an image-caption similarity threshold. This work also proposes adding image augmentations in the CLIP pretraining pipeline, namely a stacked RandomAugment strategy, and sees performance improvements in certain settings. This work primarily focuses on gathering insights in settings with small training data, small models, and few training iterations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Suggestion 1:*\nFigure 2a: increasing training epochs for smaller datasets does not significantly decrease performance, but increasing training epochs for larger datasets is important. However, I actually observe the opposite trend in this figure. Relative performance drop on the 400M datasets is about 9% whereas relative drop on the 25M datasets is about 21%. What I do see is that their performance plateaus from 16 to 32 epochs, but this is not the case for the 40M dataset. I would suggest finding a better way to illustrate/visualize this result.\n\n*Suggestion 2:*\nText at the bottom of Page 4 stops suddenly and does not continue into page 5. Similarly, on Page 5, the text in the first paragraph stops suddenly, and the conclusions from Figure 5 (b) are never discussed. Please correct these issues.\n\n*Suggestion 3:*\nFigure 6 (c)/(d) regarding data quality is particularly interesting, however many details are absent from this experiment, such as the number of iterations which the models are trained for. I would expect this specific hyper-parameters have a significant impact on the observations. For instance, with much longer training schedules would all methods perform similarly? Are these observations mostly concerning short training schedules, in which case, the observations are that high quality data yield faster convergence, but not better performance? Or do the models trained on smaller filtered data end up performing better with longer training as well?\n\n*Suggestion 3:*\nObservation that larger datasets yield better performance, even when the number of iterations is held fixed is interesting; which is mentioned at the start of page 7. But which experiments support this result, as such a conclusion is not evident from Figure 8, which trains for one epoch on each dataset size, implying that larger datasets training for more iterations. I believe Figure 9 a page down may be demonstrate this result, but I would suggest clarifying this in the text.\n\n*Suggestion 4:*\nIt is claimed that ConvNeXt has not been explored, but I would like to point out other the authors that the ConvNeXt has been thoroughly explored in OpenCLIP. Can you please clarify where that exploration falls short?\n\n*Suggestion 5:*\nConfused by Figure 16 (a) which seems to contradict figure 1b and the claim that CILP+data aug outperforms CLIP. In figure 16 (a) vanilla CLIP outperforms CLIP + DataAug for all randomly sampled data scales. I would also expect the ability to get improvements in performance to depend heavily on the data; as aggressive image transformations can result in high misalignment between the caption and the image. This also contradicts the observations in Figure 17. Can you please clarify these points, and provide more insight into this relationship?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the reviewers mentioned, the experiments are extensive and provides useful insights to understanding the behavior of CLIP. The empirical findings reported in this paper can be useful for future efforts on training CLIP-like models.
Decision: Accept (poster)