Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Authors: Thao Nguyen
Gabriel Ilharco
Mitchell Wortsman
Sewoong Oh
Ludwig Schmidt
Venue: NeurIPS 2022 Accept (2022)
Keywords: Distribution shift
machine learning dataset
image-text pretraining
robust machine learning

Abstract: Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes. In this work, we introduce a testbed of six publicly available data sources---YFCC, LAION, Conceptual Captions, WIT, RedCaps, Shutterstock---to investigate how pre-training distributions induce robustness in CLIP. We find that the performance of the pre-training data varies substantially across distribution shifts, with no single data source dominating. Moreover, we systematically study the interactions between these data sources and find that mixing multiple sources does not necessarily yield better models, but rather dilutes the robustness of the best individual data source. We complement our empirical findings with theoretical insights from a simple setting, where combining the training data also results in diluted robustness. In addition, our theoretical model provides a candidate explanation for the success of the CLIP-based data filtering technique recently employed in the LAION dataset. Overall our results demonstrate that simply gathering a large amount of data from the web is not the most effective way to build a pre-training dataset for robust generalization, necessitating further study into dataset design. Code is available at https://github.com/mlfoundations/clip_quality_not_quantity.

Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Thao Nguyen University of Washington thaotm @cs. wwshington.edii
Gabriel Ilharco
Mitchell Wortsman
University of Washington
University of Washington
gamaga satigga@@csswasinggtoo. washington.edu
mitchnw cechhh/w@sss..atieeeeee wws.swshingto.edi
Sewoong Oh
Ludwig Schmidt
University of Washington
University of Washington,
sewoong cs. catecs..waaae washington.edu
Allen Institute for Artificial Intelligence schmidt cs. washington.edi
Abstract
Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes. In this work, we introduce a testbed six publicly available data sources- Conceptual Captions, WIT, RedCaps, Shutterstock- to investigate how pre-trane tre-training distributions induce robustness CLIP. We find that the performance of the pre- training data varies substantially across distribution shifts, with single data source dominating. Moreover, we systematically study the interactions between these data sources and find that combining multiple sources does not necessarily yield better models, but rather dilutes the robustness of the best individual data source. We complement our empirical findings with theoretical insights from a simple setting, where combining the training data also results in diluted robustness. In addition, our theoretical model provides a candidate explanation for the success of the CLIP-based data filtering technique recently employed in the LAION dataset. Overall our results demonstrate that simply gathering a large amount of data from the web is not the most effective way to build pre-training dataset for robust generalization. necessitating further study into dataset design. Code is available at htttp:///tthub..coo///ooomaanooo quuiilugiioniiciiuuuulltttt not not_quantity
LAION,
Introduction
Large models pre-trained web-scale datasets becoming a comeestone of machine learning. For instance, the past two years have witnessed the arrival of several new models such as GPT-3 [2]
Chinchilla (33, and PaLM [1] for natural language processing, CLIP 48 BASIC 46 and Flamingo
3 for computer vision. These models exhibit unprecedented generalization capabilities in zero-shot inference, in-context learning, and robustness distribution shift. A key ingredient enabling their generalization performance are the large and diverse pre-training corpora that exceed previous datasets by multiple orders of magnitude. For instance, the training set of BASIC [46] contains 6,6 billion images, which is more than 1,000 times larger than the widely used ImageNet 18 training set from the 2012 competition containing 1.2 million images.
Despite the central role datasets play for pre-trained models, little is known about them, especially for image-text models. The aforementioned CLIP (48). BASIC [46), and Flamingo 3 all rely datasets internal to the respective organizations, which is also the case for other models such as DALL-E [49], Florence [63], and ALIGN (35). In addition, research publications often provide little details on the data collection processes, e.g., the data sources or data filtering mechanisms. Beyond clear issues
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

such as reproducibility and the potential presence of harmful content, the opaque dataset creation practices also make it hard to identify effective methods for assembling pre- pre-traiming datasets. As a result, researchers cannot build on each other's s dataset innovations, which obstructs the incremental research process that has successfully accumulated algorithm and architecture improvements in machine learning models. A more principled understanding of dataset creation will likely enable further progress in the generalization capabilities of pre-trained models.
A basic approach to dataset creation would be to simply train on all available data of a given type. While scaling up training sets has indeed been integral to the recent progress large models, advances in weak and self-supervision [14 [2 30 12] 20 have led to an abundance of potential training data. For instance, the large LAION- dataset [2 of 5 billion image-text pairs itself subset of about 50 billion images from Common Crawl. This abundance is already exceeding the amount of data models can currently be trained on within a reasonable time I making the aforementioned baseline of using all available data infeasible. Consequently, deciding what data to train on is becoming increasingly important.
In this paper, we take a step towards a better understanding of pre-training data and investigate the impact of dataset design on the generalization capabilities of image-text models. Specifically, we focus on CLIP, the first large model of this kind that demonstrated remarkable robustness to multiple challenging distribution shifts. The main questions we ask are: (i) How much do different web data sources vary in their induced robustness? (ii) Do dataset combinations lead to better robustness? (éii) Can filtering with an existing image-text model improve data quality? We address these questions from both an experimental and theoretical perspective.
On the experimental side, we begin by assembling a corpus of six different datasets from the web, spanning variety of sources including Flickr, Shutterstock, Wikipedia, Common Crawl, and Reddit. We then measure the robustness of CLIP models trained on each dataset. In particular, we compare the zero-shot accuracy of these models on ImageNet and a set of canonical ImageNet distribution shifts. We find that the robustness induced by each pre-training dataset varies widely, and that sources with careful curation such as Wikipedia do not necessarily outperform those with minimal filtering.
In addition, the datasets cannot be compared along single dimension: different datasets help with robustness to different distribution shifts. This in turn motivates studying how combining datasets affects robustness. We find that model trained two datasets does not inherit the robustness properties of both. Rather, while the model is exposed to both distributions, its robustness interpolates between that of the individual datasets. This indicates that dataset designers must carefully combine the pre-training data in order to preserve and enhance the robustness of the resulting models.
Building our empirical results, we introduce a theoretical model to better understand our ex- perimental findings. Our model is simple enough to allow for mathematical analysis and can still capture some phenomena of real-world, web-crawled datasets. Specifically, our theoretical model also shows that combining multiple datasets dilutes the robustness of the better data distribution. Moreover, our model provides a candidate explanation for another interesting phenomenon in the curation of pre-training data: the LAION-400M experiment (52) and follow-up CLIP reproductions [2] demonstrated that filtering noisy source (Common Crawl) with CLIP model results in dataset on which newly trained CLIP models exhibit higher robustness to some distribution shifts.
Paper outline. We first briefly review related work and the relevant background robustness to distribution shift (Section 2. before discussing our experimental setup in Section 3 Sections f and then measure the robustness of CLIP induced by individual data sources and their combinations. To support these empirical results, Section pressett our thcoretical analysis. We conclude with future research directions in Section 7
2 Background & Related Work
Vision-language models. Large vision- vision-language models like CLIP and ALIGN have become an
active area of research owing to their success on various computer vision tasks [4 35). Existing work expands their capabilities by either increasing model and dataset size [46). or by using additional supervision as in DeCLIP (40). SLIP (44), and FILIP [62). In contrast, we study the effect of pre-training data composition task performance with focus on robustness.
'Rcaal that the training set for Google's largest image-text model ALIGN contains "only" 6.6 billion images. making it about eight times smaller than the source dataset for LAION

---

In a recent work, Fang et al. [22] showed that CLIP's robustness primarily stems from its diverse pre-training distribution and not training set size, language supervision. or contrastive loss functions. However, Fang et al. [22] only conducted experiments with two datasets (YFCC-15M and ImageNet- Captions), one of which contains less than one million images. We take the insights of Fang et al. [22 as our starting point and expand the range of pre-training datasets to six different sources, each containing at least five million images. This enables us to study the robustness induced by various pre-training sources, and how dataset combinations affect robustness.
Robustness to distribution shift is a long-standing issue in machine learning [58 47 and has recently received renewed attention as researchers scruti- nize the generalization performance of neural networks in greater detail [55
Distribution shift.
45-
Imaget mmageNeta (top-1. %)
Linear fit limageNet models
Linear (LAION models)
25
8 9 3 51 27 16 38). Similar to CLIP [48], we focus robustness to natural distribution shifts, where the corresponding test sets contain only unmodified images and are not intentionally perturhed by adversarial examples or synthetic corruptions (such as Gaussian noise test robustness on ImageNetV2 150 ImageNet-R [32). ImageNet-Sketch
Effective robustness)
15-
or blur patterns) 31.. Specifically,
20
30
40
50
60
70
ImageNet (class-uubsappled) (top-1, %)
[59], and ObjectNet 6 because prior work has established many baselines for these distribution shifts [56]. More- over, models robust to these shifts also show improved robustness on other out-of of-distributiion benchmarks such as WILDS 36 60
Figure 1: Models pre-trained on LAION exhibit effective robustmess [56] com- pared to standard models trained on Ima- geNet. Effective robustness is defined as movement towards classifier which is robust to distribution shift. classifier is more robust the closer it is to the line. A classifier on the y = x line not affected by the distribution shift.
The robustness literature usually discusses distribution shift in terms of "in-distribution" and -ittoff-iittibuution" data. This terminology is natural when data from the same distribution as the "in-distribution" test set is used for training or fine-tuning, However, it is unclear what counts as "inddistribution" when models are trained on large-scee aurge-scale, generic pre-training datasets that aim to improve performance on a wide variety of tasks. To address this issue, we follow the more flexible definition of distribution shift employed in prior work [56] 43 and measure robustness as accuracy difference between two related but distinct test distributions (e.g., ImageNet and ImageNet-Sketell
The expectation is that the shift between the two test distributions should not affect an ideal robust model, for instance because the shift does not affect the accuracy of humans labelers
(33) Taori et al. (56) defined this notion of robustness as effective robustness. Radford et al. 48] adopted the effective robustness framework in the evaluations of their CLIP model. Effective robustness is illustrated Figure | and measures movement towards a perfectly robust classifier which is not affected by the shift between two test distributions.
Prior work 43) has evaluated several hundred models and demonstrated experimentally that changes to model architecture, training set size, training algorithm, and other model-related factors do not change effective robustness most cases. In contrast, changes to the pre-training distribution can improve the effective robustness of a model. This makes effective robustness useful metric for evaluating the influence of pre-training data sources on robust generalization, since removes confounders stemming from model hyperparameters and the number of training samples. The effect of the pre-training distribution on effective robustness is particularly pronounced when models are evaluated in a zero-shot setting: while training on a target distribution can improve accuracy on 61], this process can deteriorate robustness to distribution shift
that distribution (60) 255
48 60 4 46
The existence of a universal linear trend for accuracies on a pair of test sets has been analytically studied in 43 For a simple binary classification model similar to our Assumption I (see Section 6
convergence to a linear trend is shown with the deviation from the line scaling as ( Vd). However, the analysis in 4 is restricted to specific stochastic distribution shift for the -iittrriittiittnonn test data and fixed classifier independent of the training data. Hence the dependence of the slope on the training set size, variations the training methods, and properties of the training distribution cannot be described in their model. We provide significantly more fine-grained analyses (Theorem[) T that captures all such tradeoffs, which allows us to draw novel insights into how data mixing (Theorem and filtering (Theorem 3) affect model robustness.
3

---

Experiment Setup
48 which has demonstrated unprecedented zero-shot perfor- mance wide range of downstream tasks, as well robustness various distribution shifts [43]. Given an image-text pair, CLIP is trained to maximize the cosine similarity between the embeddings of the text and that of the image, relative to the similarity of unconnected image-text pairs. We use the CLIP implementation from the OpenCLIP GitHub repository (4), with ResNet-50 [28] as the image encoder model. We vary the pre-training obtain different accuracies on each data distribution. Due to compute constraints, the total training set size is most 15M samples for most of our experiments. Appendix B contains more training details.
Model. We focus on the CLIP model
size and hyperparameters such number of epochs to
Data. To study the effects of training distributions on robust generalization, we collect several datasets from publicly available sources. Most of these have been studied in the context of various vision and language tasks in previous work:
YFCC: We experiment with the 15M subset of the YFCC100M dataset 57 that the original CLIP paper 48) used for dataset ablation studies. The images and captions are collected from Flickr.
LAION [52] The images and corresponding alt-texts come from web pages collected by Common Crawl [ between 2014 and 2021. We randomly select subset of 15M samples experiment with, and ensure that the accompanying NSFW tags of all chosen images are 'UNLIKELY'.
We use CC-12N for our experiments, which consists of images and HTML alt-text from an unspecified set of web pages.
Conceptual Captions
13]
RedCaps [9] This dataset contains 12M examples, obtained from 350 manually curated subreddits between 2008 and 2020. The subreddits are selected to contain a large number of image posts that are mostly photographs and not images of people.
Shutterstock: 15M images and captions were crawled from the Shutterstock website in 2021.
WIT [54]: Image-text pairs come from Wikipedia pages. We use reference description the source of text data and obtain 5M examples after filtering to include only English language examples.
Appendix A. contains an analysis of image and text statistics, as well as randomly selected data samples from each source.
Evaluation. We pick ImageNet as the reference distribution and evaluate CLIP on four natural distribution shifts derived from ImageNet: (i) ImageNet-V2 (50)
reproduction of ImageNet test set using the original dataset creation process, (ii) ImageNet-R 32 -renditions (e.g., sculptures, paintings) for 200 ImageNet classes, (iii) ImageNet-SSetch [59]
sketches of ImageNet class objects, and (iv) ObjectNet ( test set of objects in novel backgrounds, rotations and viewpoints with 113 classes overlapping with ImageNet. Visualizations of random samples from each distribution shift can be found in Appendix A.2
4 Individual Pre-training Data Sources
We first investigate how well a CLIP model trained each data source would perform under different distribution shifts of interest. As seen from Figure 2
while all sources yield the same linear trend on ImtgeNet-V2, some display clear advantages when evaluated other test distributions. For example, Shutterstock offers the best -iitr--ddittibutionn performance on ImageNet-Sketch. while RedCaps displays the most effective robustness ObjectNet. On ImageNet-R, LAION, CC12M and Shutterstock seem to do much better than the rest. Overall no pre-training data distribution is consistently the most robust across all evaluation settings.
We also measure the data efficiency of each source, i.c., how much the performance would change with more samples from the same source, see Appendix C Similar to the previous observation, the six pre-training sources display vastly different data efficiency depending on the distribution shifts of interest. Although LAION and CC-12M exhibit similar effective robustness in all evaluation settings in Figure 2 this analysis reveals subtle differences between these two training distributions in the low- regimes.
Combining Data Sources
In the previous section, we demonstrate the variability behavior of different data sources based on the distribution shift at test time. A natural question then arises from this observation: does combining

---

-1
35
20
ImageNett22 (top-)
ImageNot.R (top-1
15
ImageNe:
75
ImageNet (top-1,
ImageNet (class-ubsaapledd) (top-1, %6)
ImageNet (top-1,
ObjectNet Rtp-1. %6.
y=x
imageNet
Liner fit (standard ImageNet models)
YFCC15m
(YFCC15m))
Linaa fit (LAION15m)
RedCaps12m
(RedCap112m)
CC12m
Linear fit (CC12m)
WiTSm
(WITSm)
ShutterStck115M
Linear fit (ShutterStock15M)
15
05 59
ImageNet (cass-subsampled) (top-1, %6
Figure 2: Performance of the six pre-training data that behavior-both in terms accuracy and slope of the linear trend-of the pre-training data varies substantially across distribution shifts, with no single data source dominating. Most shifts help highlight the strengths and weaknesses of different data sources, except for ImageNet- V2, where the linear trends produced by individual sources highly correlated with another.
under various distribution shifts. We find
ImageNet Sketch [top-] %6
Unear fit (YFCC15m)
Imagelet: (top-1
Linear fit (LAION15m)
25-
Unner fit (FFC155m-LAION15))
Linear fit (YFCC7 (0C.7755m44A007755555 5m)
1.
YFCC15m
LAION15m
YFCC1 5
YFCC7. YF0C7.5m- 5
25
55
65 75
30 40
ImageNet (clas--ubbsapleed) [top-1,
ImageNet (top-1,
Figure 3: Combining YFCC and LAION training data in equal ratios produces models with intermediate robustness. Given a fixed data budget of 15M samples, the linear trend produced by training CLIP on YPCC-LAION data mixture, with 7.5M datapoints from each source (cyan line), lies between that of training CLIP on YFCC (blue line) and LAION (green line) entirely. Even when increase the total training size (30M) use data available from both sources (orange line), the same pattern persists.
multiple sources help improve robustness across some, if not all, test distributions of interest? We investigate two common approaches of aggregating information from different training sets-input mixing and output ensembling. In the subsequent discussion, focus distribution shifts that bring out significant differences in behavior across the pre-training data sources (e.g., ImageNet-R and ImageNet-Skette Sketch); the full results on all distribution shifts can be found in Appendices) and
5.1 Input Mixing
In input mixing, randomly select and combine samples from multiple data sources to build the pre-training dataset. Our findings indicate that this exposure to more training distributions doesn't t help CLIP take advantage of the complimentary strengths of each source. Rather, the effective robustness of the resulting model less than that training on the best individual source for each distribution shift.
start with combining data from two largest data sources in the testbed -FCC and LAION. To remove dataset size as confounder, we fix the total amount of training data at 15M samples, and sample 7.5M image-text pairs from each source. This data mixture yields linear trend that lies in between the trends obtained from training on 15M YFCC and 15M LAION datapoints separately
5

---

y=x
Unear fit (LAION15m)
15
ImageNet Sketch (top-1 %6
31
Unear fit (VFCC3++LA/ONL2))
ImageNet R (top [%)
Unear fit (YFCC3M+LAIONOO
35
Uneer fit (VFCC7.5mHLAOONT55)
Unear fit (VFCC10m-LAION5m)
25
Uneer ht (VFCC12m-LAION3m)
Uinea fit (YFCC15m)
1.
LAION15m
YFCC3m LAION112m
YFCCSm + +AIONIOm
YFCC7. S0C7755m-LAION.55m
YFCCI0m LAIINN5m
25
30
YFCC12m=LAION3m
ImageNet (class (class-subsamplle subsampled) [top-1, %)
ImageNet (top-1, %)]
YFCC15m
Figure 4: Varying the sample contributions of YFCC and LAION to the input data mixture produces smooth interpolation of the linear trend between those of training on YFCC and LAION separately. Keeping total number of training samples fixed at 1SM, as we vary contribution of YFCC to the dataset mixture from 15M (i.e. only training on YFCC) to OM (i.e., only training on LAION), the resulting linear trend gradually shifts from that of YFCC-15M (blue line) to that of LAION- 15M (green line).
Image Net Sketch [top 56)
y-x
ImageNes-R ttap11 %)
(YFCC7.5m)
Liner (LAIDN7.5m) 5mm)
Liner ft (YFCC7.5M 5 5m)
VFCC7.5m+LWON/ ensemble YFCC7.5m
LAION?
YFCC7.5m LA ONT 5m
YFCC7 SmeLAOON7 5m ensemb e (epoch
PFC7755mLLOONT ensemb e (opoch B)
WCCC7.5=LUWNN 5m ensembe (epoch 1)
18 ImageNet (top-1,
imageNet ass--uusaampees ed) (top-1, %6)
Figure 5: Ensemble outputs of CLIP models trained on YFCC and LAION separately share the same linear trend as a single model trained on the combined data mixture (where each source contributed equally). We ensemble the logit predictions of YFCC-trained (blue line) and LAION- -rainned (green line) models taken from the same epoch, with varying ensemble weights between 0 and (red dashed linc). When the outputs are combined with equal weights (red markers), the resulting test accuracies closely track the linear trend produced by pre-training CLIP on a data mixture with equal number of samples from each source (purple line).
Even when remove the constraint on the training set size and use all the data available from these sources (i.e., 30M samples), the same observation on robustness holds, and the resulting linear trend is highly correlated with that of training the YFCC-7.5M + LAION-7.5M mixture.
(Figure D.
The previous set of experiments combines YFCC and LAION data with a 50:50 ratio. In Figure 4 we find that when this ratio is varied within a fixed budget of 15M datapoints, the robustness linear trends of the corresponding mixtures form a smooth interpolation between the linear trends of training on 15M YFCC and 15M LAION samples separately. Experiments with different combinations of sources, as well as mixtures of a larger number of sources, can found in Appendix [D] There we also include results for combining data from CIFAR- 10 and CINIC- 10 distributions to train ResNets image classification tasks, where we observe the same interpolation pattern. We provide a thcoretical justification for this phenomenon in Section 6.2
5.2 Output Mixing
Another common approach to take advantage of different training sets is to combine them at model output level (i.e., ensemble) [21 39 26, 23 II 24 45). Here, we train CLIP model on each pre-training distribution of interest and combine the logit predictions of all the resulting models with equal weights. In experiments with mixing YFCC and LAION (Figure 5 the ensembles of pangle-soure--tran
CLIP models taken from the same epoch of training. lie on the linear trend of single CLIP model trained on combined data. This holds across different distribution shifts that we consider. The same observation also applies when we ensemble 6 CLIP models trained separately on the 6 data sources collected, with the contribution of each model being weighted equally. Refer to Appendix for more details.

---

InvogeNve Sketch (top-], %0
ImageNetta (top-) %)
(YFCCSm-1LAIOSSM)
Unner fit (CC5ml)
YFCC5M.LAIONSM + CCSm ensomble
PRCCSm-LAIONSm
28-
VRCG5m-LAIONSm=4055
"PCCSmLLAIONSS CC5m ensemble (epoch
#FCCSm.LAIONSM CC5m ensemble (epoch 8)
PPcC3m.LAIONS0 + CC5m ensemble (epoch 4)
18
50 ImageNet (class-Subsampled) [top-1,
26 ImageNet (top-1,
Figure 6: Using ensemble outputs to predict the linear trend of input mixing without retraining CLIP from seratch. A generalization of the observation made in Figure 5jss that given an existing pre-training dataset that could be mixture (e.g.. YFCC-SM + LAION-SM, green line) and new data source (e.g.. CC-5M, orange line), we could on these two data distributions, to estimate where the linear trend for CLIP model trained on data would lie (purple line). This removes the need to actually train CLIP from scratch on the now bigger 3-source mixture.
the ensemble oulputs (blue markers) of two CLIP models that have been trained separately
We next show that this phenomenon extends to more complex mixtures. The fact that output mixing (i.e., ensembling the predictions of CLIPs trained on individual sources) predictive of the linear trend produced by input mixing (i.e., training on all the data from these sources) presents an opportunity to estimate model robustness given new data sources, without having to train the model from scratch on the new, potentially much larger, combined dataset. For example, in Figure o assuming access to CLIP model trained on the YFCC-5M + LAION-5M mixture, and another one trained on CC-5M, the ensembles of these two models across different stages of training share the same linear trend as a CL.IP model trained on the YFCC-5M LAION-5M + CC-5M mixture. In Appendix
B we show the results of ensembling 6 CLIP models trained with the 6 data sources we collected, as well as the CINIC- 10 + CIFAR- ensemble, uni-modal image classification setting where output mixing accuracies are also predictive of the linear trend of input mixing.
6 Analysis under Simple Binary Classification Models
Empirically we observe, for e.g., in Figure 2 that for a pair of test datasets (D,, D2), the correspond- ing test accuracies after probit transform achieved by various trained models (including different architectures and training algorithms) on the same training dataset D all lie on the same line. Further- more, this line includes models trained only a subset of D [43). In other words, there is universal line that is determined only by the training data distribution and the two test distributions, and is independent of which architecture from the training set. To explain this phenomenon, we study the following class of trained models parametrized by (0 Rd P E IR,,) representing the training distribution, with representing the training data size, and & E R, representing the variations due to the training algorithm.
use, how train the model, or how many samples
6.1 Universality of Accuracy the Line for Binary Classification
We analyze simple but canonical binary classification example where each training data is parametrized by its ground-truth linear classifier 0 E Id and its Signal-to-Noise Ratio (SNR) p² 4. Concretely, training dataset Dn,0,p (arq E Rdd We E +1) is set of n i.i.d. paired samples from joint distribution Po.p defined as follows.
Assumption 1 (Data distribution). We define joint distribution (2i, Pe.o as (i) y: +1 uniformly at random, and (ii) x; = y + (110 1/pp; where the noise 2, is zero-mean and has independent entries with variance one. For observation (I; yt), refer ||0|[2 as the signal power and / p2 as the corresponding noise power with SNR p²
Assumption 2 (Trained model distribution). We consider (rundom) linear model parameter in_g ddd that predicts binary label sign( (2r, 0m.g)) for test example Rdd. We assume that the random model Ô.,5 = 0 + (p/))) E Rd, trained on Dn,9.o is unbiased, El(nne) = 0, and that = has independent entries with variance one each. The randomness comes from the training data as well as any internal randomness the training algorithm. The parameter E R4 captures the variations in the resulting model distribution due changes the training algorithm.
7

---

Concretely, one canonical example of a trained model is On.s = (1/n) Ex=1 yes, It follows that On = 0 ((00)//o//)) with ç = 1. Hence, & measures the randomness of the trained model relative to this simple training algorithm.
We analyze the resulting accuracy when evaluated on two test distributions Po..pr and Poz.p= as defined above, with Acco,.e1 .e == Po (sign((X, ((.,,,,.))) = Y), and Acco2.pp defined similarly. In particular, we are interested in how the accuracy pair ( ( (AAco) - )) behaves as we vary the sample size and is the inverse of the CDF of a standard Gaussian distribution, defined as (b()) = P(= S t) where (0, 1). Φ-1 is also called the probit function. This choice of mapping the accuracy with the probit function is critical in getting the linear relation, which we will explain in Remark 1.
vary the training algorithm represented by &. Here, 1b [0, - R
Theorem 1 (Universality of accuracy on the line). Under Assumptions and 2 asymptotically as the dimension d grows linearly in the sample size n such that lims-x n/ d = a we have
Tim Acco = (b cos((0,, 0) pipcr) and Tim Acco2,22 = $ coss( cos((2,, 0) 02p0r d-+00
(1)
d-000
g
Further, under Assumption in the Appendix, for some universal constant c > 0,
cos 0)p x (AAco.. + 0 cos(O1, p1
Φ - (Acco2.p2)
(2)
We provide a proof in Appendix F.1 This analysis implies that for any training sample size n and any (variation due to the) training algorithm &. the resulting accuracy pair after probit transform lies on a universal line determined by Eq. @ and the slope only depends on the two test distributions (Por. Po2 and the training distribution Po.p- The similarity between the training data and each test dataset is captured by the angles: cos(01, (f1, 6) and cos(62, 0). More similar training data achieves a larger test accuracy. The hardness of the test distribution captured by the SNR of each test dataset: P1 and P2- We emphasize that this line is universal in the sense that the slope does not depend on the sample size n and training algorithm parameter g. We are interested in the regime where n scales linearly in d and both are large such that the second term in the above equation is negligible.
Why do we get the accuracy-on-the-linee (random) linear model on random test data point X involves the inner product, which performs a natural spatial averaging over the d coordinates. Since the noise across the coordinates is independent and has bounded variance, the central limit theorem applies. The resulting error has a Gaussian tail. Hence, the probit mapping, - (Acco, ), is critical in translating accuracy into the relevant mean to standard deviation ratio: cos(@1. ((1,00))ppp (D)pp///((v/)
Remark 1.
phenomenon? The prediction of the trained
This is consequently important for getting the universal linear relation, because irrelevant parameters, and E, cancel out in the slope of Refer to the proof of Theorem [ (Appendix F.1 for more details.
x --1 (Accoge))))) /
Remark 2. Intersection at the random guess: Previous work 43 that considered models with a wide range of accuracies has found that all lines intersect at a point corresponding to "random guess", which in this binary example is (1/2), (1/2)) = (0, 0). Our theoretical analysis is consistent with this empirical observation: all universal lines intersect at (0, 0) up to a small additive error scaling as (1)
6.2 Input Mixing Yields an Intermediate Slope
Figures 3 and show that when a model is trained on samples combined from two datasets, the resulting robustness trend lies in between the trends achieved by individual datasets. We show that this finding is universally true under our current setup in Theorem Note that for the linear models we consider, input mixing (combining training sets) and output mixing (combining model outputs) are equivalent.
Assumption 3. Consider two training datasets D D 8 and D 82.02 of sizes n4 and n2 from distributions Por.p, and P P defined in Assumption Separately training on individual datasets Dn,,0 on a combined (and possibly subsampled) training dataset Dns, Dni. UD. 82.02 represented by
gives two models fn. (D by .... and &rez (D, 02..8..p2 .2) as defined in Assumption
2 A model trained
Ong Pni,nihe (P, 20.02 U D Ds) 0 a+ (E110) n)))² where 0 = (n, 0, + n2/02)/(())
8

---

( (m/ 111 + (nb110211//P)), and 2 E Id is a zero-mean random vector with independent entries each with variance
m²),, p - 181//
Again, canonical example of trained model is 0n; On + (1/(n) (E y:2.T
P... 62..6... .ag y: We then have Ung+n 0 + (11011/ (aell/O@v// (Dvn' + nl)) with ==1.
It follows from applying Theorem Io the model trained on the combined data (Assumption) 3 that the resulting linear trend achieves
cos(02, 0)2
Slopo(On) +ni (Dr,, 01 UDri,
cos(O,, 0)pt
We show that the slope obtained from training the combined dataset lies between those obtained from training on the two datasets separately. The proof could be found in Appendix F.2
Let
Slope, == Slopo(On, (D 01 )) and Slope, be defined similarly.
Theorem 2. Under Assumption Slope; I Slope( (rope(Ox,+aa)) Ong (D (Dnj,0ra UD Uni Di..00.02)) S Slope
6.3 Filtering Data to Improve Robustness
The input mixing analysis in Section 6.2 suggests a filtering strategy to improve robustness.
Assumption 4 (Gaussian distributions for filtering). Consider a training dataset D Dn,0eramo of size n. draw i.d. from (X,Y) ~ -Pocass where Y = =+1 uniformly at random and X|V
(Y Orrains (1)Orrain p)) IT) A model trained on an unfiltered Dn.Puraius is denoted by Ountittered (L/n) E (e,,)))DD.... Oeraimo Jzyh. Note that Ounfiltered ~ (Otrain (10rrainll (pvll//
The model is evaluated on two test datasels: in-distribution (ID) and out-of-ddittribution (OOD), each with an isotropic Gaussian noise: (X,Y)
PoID.PD where Y = +1 uniformly at random and X|Y NY (1OODll/ II), and PooOD.POOD is defined similarly. Let Slope(0) cos(BooD. d'pooon and assume that all models achieve better ID accuracy than OOD accuracy, i.e., cos(0rn.00ym Sl)pe(O) < 1. A model is more robust if the slope is closer to one.
Suppose have access pre-trained model 0 pretrained that achieves better robustness than the model Ō unfilteree trained on the unfiltered data, i.e., Slope(0unfittrre)) < Slope(0 pretraieed) S 1. We want to use the pre-trained model to filter the data Dn. and achieve better robustness. We consider a generic family of filtering strategies that subsamples each data point (Ii, M4) E DD, with probability that is monotonically non-decreasing with its correlation to the pre- pre-trained model, i..,, ((ra, Vi) passes the filter) h((259) θ ppretrained) for some monotonic scalar function h R - R.. We analyze the model 0iiterrdd = (1/m) E(r.v. (x:..).Efillee daiaset TTy; The next theorem shows that any such filtering strategy improves robustness, and the full proof can be found in Appendix F33
Theorem 3. Under Assumption 4 if (loppe(uuniierrea) < Slope (prett(pra))ainne) S 1 then any model @nttored that is trained on the above family of filtered datasets achieves better robustness than the Slope ((iittrra))) < Slope( (lept(@pretraiinaa)
model trained on unfiltered data: Slope( @ (anpttiaitle)))))) where (lope(E[(nntereal) denotes the slope of the linear trend of a model trained on the filtered data.
Remark 3. Relevance to empirical practice: The data filtering approach currently employed for the LAION dataset 5 motivates our theoretical setup. The authors of LAION first retrieved a large number of image-text pairs from Common Crawl [I] and then computed the cosine similarity between the image and text embeddings using one of the CLIP models originally introduced by Radford et al. [48). Next, the authors removed all pairs with similarity less than 0.3 from the dataset. While it was initially unclear if filtering the noisy Common Crawl source with an existing CLIP model would lead to dataset with good generalization properties, experiments with medium-scale models on LAION so far demonstrate that the resulting models are comparable to the original CLIP models [2].
Our theorem 3 does not provide a full justification for the filtering process described above, but it offers theoretical evidence that filtering a noisy data source with an existing robust model can indeed improve the effective robustness of the resulting dataset. It remains to be shown empirically that

---

the slope of the linear trend exhibited by the CLIP model from 48 is larger than that of model trained on the Common Crawl data pool without filtering. If this assumption holds, we can expect that training on data with better text-image alignment according to the original CLIP model, will improve robustness compared to training on unfiltered data from Common Crawl.
7 Conclusion
Motivated by CLIP's robustness to distribution shift, we introduced a testhed of six image-text datasets to study the influence of pre-training data on the robustness of multimodal models such as CLIP. We analyzed the interactions of these datasets through both experiments and theoretical analyses, finding that simply combining multiple datasets dilutes the robustness of the rest-performing one. Moreover, we offer an explanation for the potential benefits of the data filtering process currently used for LAION. Our findings suggest that training on carefully selected subset of data may provide more robustness than training on simply more data, e.g. obtained by combining multiple data sources.
To summarize, our results lead to the following recommendations for dataset design:
On the evaluation front: building pre-training dataset requires multiple robustness test sets, as different pre-training datasets exhibit vastly different behaviors across distribution shifts.
On the dataset design front: each candidate data source for the overall pre-training dataset should be evaluated separately for its generalization and robustness properties. The final pre-training dataset should then contain more samples from the higher-quality pre-training sources. Creating more "diverse" pre tre-training dataset by randomly sampling from all candidate sources does not result in better dataset.
Assuming access models separately trained on each source of interest, output ensemble is good predictor of the effective robustness obtained from input mixing, and hence can significantly reduce the time to search for the right mixing strategy.
Filtering a noisy data source with an existing robust model can improve the generalization properties of the resulting dataset.
The broad societal implications of image-text models and web-crawled multimodal datasets have been extensively discussed by Radford et al. [88] and Birhane et al. [0] respectively. As these models continue to grow in size, full-scale experiments are becoming prohibitively expensive. This necessitates small, systematic experiments that allow reliable extrapolation to larger scales. Since effective robustness is independent of model size and quantity of training data, we hope that our testbed and results can serve as a first step towards building multimodal pre-training datasets in a principled manner. There remain interesting open questions about finding better ways to filter and combine samples from multiple data distributions, or how to build the most robust dataset given a fixed set of data sources. We look forward to addressing these in our future work.
Limitations. The pre-training datasets we experiment with range from 5M to 15M samples. While this lags behind current kate-of-thh-art settings that large image-text models are often trained on, we believe our findings also hold for models with higher accuracy for the following reasons:
The original CLIP paper [48] offers evidence showing that CLIP's performance exhibits reliable linear trends across compute scales (Figure 9 of [48), which includes the ResNet50-encoder that our paper works with) and performance on natural distribution shifts (Figure 14 of [48)). In particular, the linear trends in this Figure 14 are analogous to the linear trends in our scatter plots. In the OpenCLIP GitHub repository (34] that seeks to match the performance of the original CLIP paper, the authors there also note that their models that were trained on 15M samples or fewer share the same effective robustness trend as OpenAl's CLIP models (trained on 400M samples). Refer to their Why are low-accuracy CLIP models interesting? remark for more details.
Previous work 4 has experimented with training randomly sampled subsets of the training dataset to produce models covering a wide range of accuracies, and found that these models lic on the same effective robustness trend. This includes models trained on only 1% of the original data, which matches the scale of our experiments (5M-15M samples) relative to the scale of the original CLIP paper (400M samples).
The focus of our work is to study the behavior of a diverse set of data sources, and performing full-scale experiments on 6 datasets (and their combinations) would be extremely expensive. This prompts us to leverage publicly available web-crawled datasets, and study the behavior of our trained models through the effective robustness framework, which has been shown to be ccale-invariant.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Tiffany Cai, Nicholas Carlini, Tatsunori Hashimoto, Jong Wook Kim, Hongseok Namkoong, Alec Radford, Rohan Taori, and Thomas Wolf for helpful discussions while working on this paper. We also thank Tiffany Cai and Jack Gallagher for their help with the dataset collection. HuggingFace has provided generous assistance in the form of compute resources that made many of our experiments possible. particular, we would like to thank Leandro von Werra and Victor Sanh for their help with setting up the compute infrastructure and their constructive feedback on the dataset testbed. This work is supported in part by Open Philanthropy. the Allen Institute for AI, and NSF grants Institute for Foundations of Machine Learning (IFML).
IIS- 192-992 DMS-2134012, and CCF-2019844 as a part of NSF
References
[1] Common crawl. https://commoocrawll.
Accessed: 2022-05-18.
[2] Laion-5b: A new era of open large-scale multi- modal datasets. https://laion. ai/ lai on 5b-a- aob-a-new--oe era- of oof-open--e oaen-earge--ea-eeeeeaaaa large-scale-eeeeeeeeeeeeeeaaleae scale- acale-multis -oda--mmdlddaaassss 2022-05-18.
Accessed:
[3] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[4] A. Andreassen, Y. Bahri, B. Neyshabur, and R. Roelofs. The evolution of out distriibit robustness throughout fine-tuning, 2021. https: httpgi///axiv... arpiv.xiiv......... org/ abs/a////..2/////// 15883
[5] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization, 2019. https://arxis: org/ abs/1907 02893
[6] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: large-scale controlnd dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[7] E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine learning, 1999. https::/link. wtreek.rriiinne com/article/ 10.1023/A 1007515423155
[8] B. Biggio and Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317-331, 2018.
[9] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šindić, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In Joint European conference machine learning and knowledge discovery in databases, pages 387-402. Springer, 2013.
[10] A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2 10. 963, 2021.
[11] L. Breiman. Bagging predictors. Machine learning, 1996. https://1ink.sprrltn com/ article/10.1007 1007//FF000865555
[12] Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam,
G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[13) S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[14] T. Chen, S. Korbblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
11

---

[15] A. Chowdbery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[16] A. D'Aour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton,
J. Eisenstein, M. D. Hoffman, al. Underspecification presents challenges for credibility in modern machine learning, 2020. hatpss://axx/ org/ abs /201..21... 03395
[17] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey. Cinic-10 is not imagenet or c0f arXiv preprint 1r1.v::81000350: 2018.
[18) Deng, W. Dong, R. Socher, L..J. Li, K. Li, and L.. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009. https
//ieeexplore. leee. bareee.org//ooee
5206848
[19] K. Desai, G. Kaul, Z. Aysola, and J. Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111. 1431, 2021.
[20] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 181004081000805, 2018.
[21] T. G. Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, 2000. sttps://iin..springerr wtt.ccom/ccca. wwwmm/hhapeer/1.10000
3-540- 0640-45014-9-1
[22] A. Fang. G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). arXiv preprint arXiv:2205.0139) 2022.
[23] S. Fort, H. Hu, and B. Lakshminnaayaanan Deep ensembles: A loss landscape perspective, 2019. htttps://arxx wwwrrxvv..g//eee. wtggabb//9112.22/
[24] Y. Freund and R. E. Schapire. A decision-theoretie generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 1997. /ettss//wwwy sciencedirect. com/ wttp:/cceenccrraree
[25] Gao, S. Geng, R. Zhang, T. Ma. Fang. Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters, 2021. atttss://arxiv wttp:///a.xxio.oo/bbbbao//// 2110. 04544
[26] R. Gontijo-Lopes, Y. Dauphin, and E. D. Cubuk. No one representation to rule them all: Overlapping features of training methods, 2021. hntpp://arxxvvorr//aaa wwwp://or//aaa//....oooo. 00134
[27] I. Gulrajani and D. Lopez-Paz. In search of lost domain generalizationn. Int. International Conference on Learning Representations (ICLR), 2020. htaps:://axxiv.org//abaaa
01434.
[28] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[29] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[30] K. He, X. Chen, S. Xic, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[31] D. Hendryeks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. International Conference Learning Representations (ICLR), 2019. https://arxiv/ org/ abs/1903, 1903. 12261
[32] D. Hendrycks, S. Basart, Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Para- juli, M. Guo, et al. The many faces of robustness: A critical analysis of diitt--iitt generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision pages (340-8349, 2021.
12

---

[33] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya. T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprini arXiv:2203. 15556, 2022.
[34] G. Ilharco, M. Wortsman, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong. J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL 5281/zenodo 5143773, If you use this software, please cite it below.
[35] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le. Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[36] W. Koh, Sagawa, H. Marklund, M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[37] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[38] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and
A. Courville. Out-of-distribution generalization via risk extrapolation (rex), 2021. https /arxiv.org/abs/////. wrg//bb////bbs////// 00688
[39] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2017. https://arxivv org/ abs/abss/an 1/1//11121...144
[40] Y. Li. F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint 10.00221 2021.
[41] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:1608.03398, 2016.
[42] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2017.
[43] Miller, R. Taori, A. Raghunathan, Sagawa, P. W. Koh, V. Shankar, P. Liang, Carmon, and L. Schmidt. Accuracy on the line: on the strong correlation between dustt--diit and in-distribution generalization. International Conference on Machine Learning, pages 7721-7735. PMLR, 2021.
[44] N. Mu, A. Kirillov, D. Wagner, and S. Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint arXiv:2112.1275:: 2021.
[45] J. Nixon, B. Lakshminarayanan, and D. Tran. Why are bootstrapped deep ensembles not better? In "I Can't it Believe It's Not Better!" NeurlPS 2020 workshop, 2020. URL https //openreview
[46] H. Pham, Z. Dai, G. Ghiasi, H. Liu, A. W. Yu, M.-T. Luong, M. Tan, and Q. V. Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.1000 2021.
[47] J. Quinioneo-Cannella, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning. Mit Press, 2008
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[49] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss. A. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[50] B. Recht, R. Roelofs, I.. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
13

---

[51] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2019.
org. htt///bb//11111 191110871111
[52] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarezyk, Mullis, A. Katta, Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, arXiv preprint urXiv:21 2021.
[53] V. Shankar, R. Roelofs, H. Mania, A. Fang, B. Recht, and L. Schmidt. Evaluating machine accuracy on imagenet. International Conference on Machine Learning (ICML), 2020. http://proccedding" mlr. press/vi19) 20/ aataaar//00/asaaaaaaaaaa shankar pdf
[54] K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork. Wit: Wikipedia-bass based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443 2443-2449, 2021.
[55] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:/312.6199 2013.
[56] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18899, 2020.
[57] B. Thomee, D. Shamma, G. Friedland, B. Elizalde, Ni, Poland, D. Borth, and L.-J, Li. Yfce 100: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[58] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[59] H. Wang. S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[60] M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Korblith, R. Roelofs, R. Gontijo-Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, and Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint wxXi:2219.00193 2021.
[61] M. Wortsman, G. Ilharco, S. Y. Gadre, Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple ine-tuned models improves accuracy without increasing inference time. International Conference Machine Learning (ICML), 2022. https: https://arxiv org/ abs /aab///22233 05482
[62] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu. Filip: Fine -rreeegraiddddd interactive language-imag image pre- pre-training. arXiv preprint arXiv:2111.0778. 2021.
[63] L. Yuan, D. Chen, Y.-.. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint 111.422111.11432, 2021.
[64] R. Zhang, Fang, P. Gao, W. Zhang. K. L., Dai, Y. Qiao, and Li. Tip-adapter: Training- free clip-adapter for better vision-language modeling, 2021. hhtpps///axxiv orggabs////11/11 03930
[65] K. Zhou, J. Yang. C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https //arxiv. www//bs.////eeeoooooooo.oee.
abs 2109, 01134
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see Section 7
(c) Did you discuss any potential negative societal impacts of your work? [N/A] we study publicly available data sources
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] see Appendix F.
F2 E33
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No], our models were trained using code from open_clip GitHub repository, which is publicly available (34]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see Appendix B
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
Appendix C
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see Appendix
4. If you using cxisting assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license the assets? [Yes] for datasets that have been published,
we refer readers to the corresponding publications. For Shutterstock, the license follows
what stated on the website (see Section 3)
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "wKxf3oXsd7_", "reviewer": "Reviewer_Y6Mj", "summary": "This paper studies the role of training data on the robustness of large image-text models like CLIP and ALIGN.\nThe paper is structured into two main parts — empirical analysis, and theoretical discussion.\nThe empirical analysis involves training CLIP models using six different datasets containing web-curated image-text pairs, and report their robustness on existing testbeds covering ImageNet classes.\nThe second half of the paper presents a theoretical discussion considering a binary classification setup, to explain empirical results.\nTaken together, the paper argues that naive scaling of training data is not sufficient to improve robustness -- careful consideration during data curation is critical to endow the trained models with the desired robustness.", "questions": "The experimental setup makes heavy use of the recent LAION dataset.\nA recent audit of the LAION dataset found several ethical issues with the dataset, e.g. verifiably pornographic imagery (especially of marginalized communities).\nSee \"[Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://arxiv.org/abs/2110.01963)\" for a more comprehensive discussion.\n\nThe authors have applied basic filtering to avoid NSFW content, which is a great step -- I suggest citing this dataset audit paper and making a direct note in the paper to ensure that future works also apply similar filtering to minimize over-amplification of problematic imagery.\nI also suggest releasing all trained models to aid follow-up works and increase the potential impact on the community.", "limitations": "**Discussion about limitations and societal impact is adequate (Section 7).**\nI concur with the authors' assessment – this paper focuses on analysis and understanding of CLIP (and similar models), it is fair to say that its societal impacts are not very different/more than CLIP itself.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this paper, in its current form, already matches the quality of a typical publication at NeurIPS conference. I strongly recommend acceptance. Below I justify my rating by discussing the main strengths of the paper along each of the suggested axes:\n\n**Originality:**\nLarge pre-trained models like CLIP and ALIGN are becoming increasingly popular in the vision community.\nWhile most works make progress by scaling up data and models (e.g. ALIGN, BASIC) or by adding dense supervision (e.g. SLIP, DeCLIP, FILIP), this paper focuses on the impact of training data source/distribution on certain downstream tasks.\nExperiments study CLIP's robustness conditioned on its training data, and theoretical discussion attempts to explain results obtained using noisy, internet-scale data.\nTo the best of my knowledge, this exploration using large pre-trained models at this level of detail is novel.\n\nSide note: Findings of this paper match with a recent paper — [Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)](https://arxiv.org/abs/2205.01397). However, the experiments in this paper are different and the theoretical discussion is new. Moreover, this paper appeared on arxiv very close to the NeurIPS deadline so I would consider it a concurrent work. It may be worth citing this paper later in the camera-ready.\n\n**Quality:**\nSection 1 (Introduction) lists three precise questions explored in the paper (Line 41—44).\nThe experiments and theoretical explanations are well designed to concretely answer these questions.\nAuthors do a good job at defining minimal experimental setup and executing it thoroughly.\nThey begin with a common setting of training CLIP models using one/more of six existing image-text datasets, and answer each question by showing trends among a subset of trained CLIP models. In all experiments, the authors have trained multiple models when subsampling training data to capture the variance in subsampling and training runs.\n\n**Clarity:**\nThe writing and presentation of this paper is excellent.\nAll technical details for empirical analysis are well-stated and easy to follow.\nBoth parts of the paper — empirical and theoretical — are broken down into sub-sections for the reader to easily find answers to each question asked in the introduction. All plots are neat and figure captions are self-contained.\nSome plots often have 10 or more colors in the legend — the authors have used color-blind friendly palette and differently shaped markers to improve accessibility.\nThe supplementary includes some basic statistics of data used in the experiments (caption lengths, image resolution, qualitative examples, etc.)\n\n**Significance:**\nIn this \"scaling era\", we are seeing rapid progress with large models like CLIP, ALIGN, etc. that are often trained on in-house data containing millions to billions of training instances.\nThe quality and concept distribution of training data plays a huge role in the success of these models.\nUnfortunately due to its unprecedented scale, the training data is becoming increasingly black box. The general consensus in the field states that adding more training data will keep pushing the model performance. This paper presents some contrary evidence in the context of model robustness, which is rather not obvious. Hence, I think this is a timely contribution that will force practitioners and researchers to think more in-depth about the training data distribution when scaling CLIP-like models.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vcdujq0mXN", "reviewer": "Reviewer_QdN7", "summary": "This paper investigates the effects of training data distributions for vision-language pretraining. Specifically, the authors train CLIP on six publicly available datasets (image-text pairs) individually or collectively and then measure models’ robustness on ImageNet variants. In the experiments, the robustness against distribution shifts is shown as a linear trend (line) of accuracy on the base ImageNet data and its variants. Each line is compared with the y= x line (the perfect model). The empirical results suggest that 1) no single training data provides consistent out-of-distribution performance and 2) combining datasets (each has its own strength) might hurt robustness of the model. The authors also analyze the results of ensembling model outputs trained on different datasets. Lastly, the authors provide theoretical analysis on the universal line only determined by the training and two test datasets (independent from model architectures).", "questions": "- Have you considered other tasks (e.g., image2text/text2image retrieval)? Or, do you have preliminary results on other tasks?\n- How did you decide the hyperparameters for CLIP (shown in Appendix B)? \n", "limitations": "As I mentioned in the weaknesses, the empirical results are drawn from the limited data setting, i.e., there is a significant gap between the experimental setting and the training setup of SOTA models. Also, the robustness against distribution shifts is investigated using a unimodal task (ImageNet). ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- This paper investigates the robustness of models trained on different training data configurations. This is a nice step towards understanding the data efficiency of contrastive training.\n- Overall, this paper is well-written and clearly explains the motivation, experiments, and analysis.\n\n[Weaknesses]\nMy main concern is the discrepancy in scale between the experimental setting and the current SOTA models (e.g., CLIP, ALIGN, MURAL, CoCA etc.). Therefore, I’m not sure if the conclusions of this paper hold for other vision-language models trained with contrastive loss.\n\n- Vision-language pre-training is usually done using noisy massive image-text pairs (e.g., ALIGN uses 1.8B examples). In the experiments, the size of training data (up to 15M examples) is quite different from the training setup of SOTA models. \n- Although the number of negatives (i.e., batch size if using batch negatives) is an important factor of contrastive learning, it’s unclear if the authors performed an initial investigation for it (batch size of 128 sounds relatively small). \n- The robustness of the models could be different in other tasks. One of the major tasks (for both pre-training and test) is image-text retrieval, but evaluation is done by a unimodal task only (ImageNet). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QbGxLR0HqTc", "reviewer": "Reviewer_8piD", "summary": "This work investigates how pre-training distributions induce robustness in image-text models. They use Contrastive Language-Image Pre-training (CLIP) as a case study and evaluate it with six publicly available datasets (i.e YFCC, LAION, Conceptual Captions, WIT, RedCaps, and Shutterstock). The overall results and ablation studies show that the robustness of CLIP and other pre-trained image-text models does not depend on the quantity of the training data/data sources, but rather depends on the data quality. In addition, this work shows that mixing/combining multiple data sources/training data only dilutes the robustness of the best individual data source instead of yielding better models. Finally, the authors call for further study into dataset design, citing that web-crawling is not the effective way to build a pre-training dataset for robust generalization. ", "questions": "Please refer to the weaknesses section.", "limitations": "Please refer to the weaknesses section.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written, clear, and easier to understand and follow.\n2. The studied problem is very interesting and important to the NeurIPS community, given that web-based data sources/training data are widely used in pre-training large models.\n3. The experimental and research methodology used clearly support the central claim of the paper.\n\nWeaknesses:\n1. This work does not provide the details on how the data sources were combined during input mixing in Section 5.1. Were the samples selected randomly or controlled? If randomly did you try selecting using different random seeds? How diverse were the selected samples? This also applies when selecting the subset of the used datasets in Section 3. It would be better to clarify this, since different samples may contribute differently during pre-training?\n2. It would also be more helpful to the community if you would discuss/recommend better dataset design approaches for robust generalization. As the paper title touches on both dataset design and robustness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept