On the Connection between Pre-training Data Diversity and Fine-tuning Robustness
Authors: Vivek Ramanujan
Thao Nguyen
Sewoong Oh
Ali Farhadi
Ludwig Schmidt
Venue: NeurIPS 2023 spotlight (2023)
Keywords: robustness
out-of-distribution shifts
finetuning
pretraining

Abstract: Pre-training has been widely adopted in deep learning to improve model performance, especially when the training data for a target task is limited. In our work, we seek to understand the implications of this training strategy on the generalization properties of downstream models. More specifically, we ask the following question: how do properties of the pre-training distribution affect the robustness of a fine-tuned model? The properties we explore include the label space, label semantics, image diversity, data domains, and data quantity of the pre-training distribution. We find that the primary factor influencing downstream effective robustness (Taori et al., 2020) is data quantity, while other factors have limited significance. For example, reducing the number of ImageNet pre-training classes by 4x while increasing the number of images per class by 4x (that is, keeping total data quantity fixed) does not impact the robustness of fine-tuned models. We demonstrate our findings on pre-training distributions drawn from various natural and synthetic data sources, primarily using the iWildCam-WILDS distribution shift as a test for robustness.

On the Connection between Pre-training Data Diversity and Fine-tuning Robustness
Vivek Ramanujan' Thao Nguyen" Sewoong Oh Ludwig Schmidt Ali Farhadi
University of Washington Allen Institute for Al
{thaottn, ramanv) (amnanvvacs.wwshingaae washington edu
Abstract
Pre-training has been widely adopted in deep learning to improve model perfor- mance, especially when the training data for target task is limited. In our work, we seek to understand the implications of this training strategy on the generalization properties of downstream models. More specifically, we ask the following question: how do properties of the pre-training distribution affect the robustness of fine- tuned model? The properties we explore include the label space, label semantics, image diversity, data domains, and data quantity of the pre-training distribution. We find that the primary factor influencing downstream effective robustness [4] is data quantity, while other factors have limited significance. For example, reducing the number of ImageNet pre-training classes by 4x while increasing the number of images per class by 4x (that is, keeping total data quantity fixed) does not impact the robustness of fine-tuned models. We demonstrate distributions drawn from various natural and synthetic data sources, primarily using the iWildCam-WIn WILDDS distribution shift test for robustness.
findings pre-training
Introduction
Transfer learning is a popular technique to deal with data scarcity, improve training speed, or transfer useful inductive biases that benefit downstream tasks [28 g [0]. In the domain of computer
vision, pre-training on ImageNet in particular has been the de-facto standard for obtaining features
to solve wide range of vision tasks, such as object detection 34 g [8]. segmentation (5] 16]. and
action recognition [42]. While there exists previous work that seeks to pinpoint specific properties of
the analysis is often done with respect to model accuracy. Our work instead examines the robustness of fine-tunce models natural distribution shifts. Instead of looking at architecture variations and pre-training algorithm as done in prior work [38] [1] [4),, we focus on the role of the pre- pre- vee-rraining data. This data-centric approach has been validated by past work (27 29 [3), which show that the training data distribution plays larger role than training methods or architecture in influencing model robustness.
PeageNet-trainee features that benefit downstream performance 19
23 38)
Robustness under distribution shifts is fundamental concern for producing reliable machine learning systems: a model can perform in unexpected and undesirable ways when there is mismatch between the data distribution encountered in deployment and the one on which the model is trained [2] For example, a self-driving car should be able to generalize to a wide variety of weather scenarios to be considered safe, some of which may have seen during training. In our work, focus on these forms of natural distribution shifts [33 22 named so because they are induced by real-world processes and study what aspects of the source dataset could help fine-tuned models become more robust to these shifts. We tackle this question along five different ablation axes: (i) Data quantity, (ii) Label granularity, (iii) Label semanties, (iv) Image diversity, and (v) Data sources. Through
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Pretraining
Finetuning
Evaluation
Pretraining Datasets
Test Sets
3un dirrbuse
Frstabee
Noluralit
000
Distribtion
Out of Distribution
Model
Model
Model
Figure I: A summary of our experimental pipeline. We pre-train model on variety of different data distributions and evaluate its effective robustness after fine-tuning on downstream task (i.e., iWildCam). By examining many models in this manner, pre-training distribution that are important for fine-tuning robustness.
determine empirical properties of the
better understanding of the interplay between various properties of the pre-training distribution and downstream robustness, we seek to establish guidelines for constructing better pre-training datasets for fine-tuning.
Previous work by Miller et al. 27 experimented with a wide range of natural distribution shifts and found that pre-training on ImageNet yields the biggest improvement robustness for the iWildCam- WILDS dataset [22] 22. Consequently, we use iWildCam- WILLS interventions with the ImageNet pre-training distribution would alter the robustness trend uncovered in this previous work. We also analyze the of other pre-training data sources that may differ significantly from ImageNet both semantic content and data collection methodology. Our main findings can be summarized follows:
probe to evaluate how our
(i Data quantity. Pre-training with more data helps boost robustness. However, we do not need lot of pre-training data to see significant robustness gains: using 25K images subsampled from either ImageNet or iNaturalist, which 6x smaller than the size of the fine-tuning dataset, already offers noticable robustness improvements.
(ii) Label granularity. Making labels more coarse-grained lowers transfer robustness. The effect less significant than altering data quantity: extreme reduction in label granularity (e.g., using 5 coarse classes instead of 1000 fine-grained classes) still preserves some of the robustness gains compared to training from scratch.
(iii) Label semantics. Given enough data and labels, using more semantically similar classes does not have a notable impact on the robustness of fine-tuned models. In particular, we find that pre-training on the 600 inanimate object categories in ImageNet yields the same effective robustness as pre-training on the 400 animal categories, despite the fact that the downstream task consists of only animal categories.
(iv) Image diversity. Given the same pre-training label set and data quantity, increasing per-class diversity (c.g.. by including more subclasses) has effect transfer robustness. In addition, the trade-off between having more classes and more images per class is not significant if total number of samples is kept constant.
(v) Data sources. We find that natural data sources (i.e., ImageNet, iNaturalist) yield similar downstream robustness when controlling for data quantity. Pre-training with synthetic fractal data is less effective at the same data quantity regime but still has some robustness gain to offer compared to training from scratch. Synthetic natural-looke lookkng data (c.g., generated by Stable Diffusion (35]) can help close this gap between using natural data and synthetic fractal data.
Overall we find that increasing pre-training data quantity and label granularity makes fine-tuned models more robust to distribution shifts. However, not all additional data is equally helpful. For instance, the context of WWilCCam- WILDS task, pre-training with natural-looking data offers much more robustness than using 10x more synthetic fractal data.

---

50
y-x
40
Trained scratch
InageVet presrained
Figure 2: Effective robustness is defined move- ment towards classifier which is robust to distri- bution shift (i.e., line y 2t. Using this metric, Miller ct al. [27] observes that for the iWildCam- WILDS task, models pre-trained on ImageNet are much more robust than models trained from scratch. We reproduce these two trends and use them as points of reference for our subsequent ex- periments, in which we modify the pre- pre-training distribution and observe how our interventions al- ter the robustness trend lines.
Imagelett precaened (Milr af
OOD Test macro
30
Traised from scratch (Wilere at all
20
10
Effectyce robustness
10
20
30
40
50
60
70
ID Test macro F1
Figure 3: We visualize the residuals of various architectures after fitting a lin- ear trend that predicts OOD accuracy from ID accuracy. All models are pre- trained on the full ImageNet dataset and fine-tuned on iWildCam for 12 epochs. We observe that overall the residuals fluctuate around the y = 0 line and vary throughout the course of fine-tuning for most architectures.
0.04
Trend Residual
0.02
0.00
0.02
alexnet
densenet121
-0.04
resnet18
resnet34
-0.06
resnet50
8
10
11
Epoch
2
Background
The main motivation for our paper comes from the work by Huh et al. [9] which investigates various factors ImageNet training that affect the quality of the features used subsequently for transfer learning. For our investigation, we shift the focus from accuracy to robustness against distribution shift, which has been a long-standing issue in machine learning B1] 43 analyze the robustness of pre-trained features natural distribution shifts observed in the real world through the iWildCam- WILDS benchmark [22]. Furthermore, in contrast to Huh et al. [19] we experiment with greater variety of more recent neural network architectures, in addition exploring the use of synthetic pre-training data.
In particular, we
A key goal in robustness is to reduce the impact of distribution shifts on the performance of model. model performances on in- and out-of-distribution test sets are plotted along the x and y-axes of a scatter plot respectively, then more robust model would lie closer to the diagonal y x line. This notion of robustness was captured by Taori et al. 44 under the term effective robustness, which measures the difference between model's 's actual OOD performance and what could be predicted from its ID performance (Figure 2). Miller et al. (27] adopted this effective robustness framework and evaluated hundreds of models ID performance is highly correlated with OOD performance. This linear trend mapping ID to OOD performance, and how close it is the y line, is what we in our work to compare the quality of the pre-trained features.
various distribution shift settings. The authors observed that
More notably, Miller et al. 27 discovered that on the iWildCam dataset, models trained from scratch and those that have been pre-trained on ImageNet on distinct linear trends, with the latter exhibiting more robustness. We replicate these reported trends in Figure Motivated by this result, our work seeks to better understand what aspects of ImageNet pre- teeiirrining contribute to this improved robustness iWildCam, and how these aspects translate to other pre-training data sources.
Previous work by Andreassen et al. [ has looked at effective robustness over the course of fine- tuning and found that pre-trained models exhibit high effective robustness in the middle of fine-tuning which eventually decreases as the training proceeds. The paper also experimented with ImageNet as one of the pre-training data sources. In our investigation, as sanity check to remove number of training epochs as a potential source of bias for the linear fit, we adopt the linear trend of models pre-trained on ImageNet and fine-tuned on iWildCam computed previously by Miller et al. [27] as the baseline. We then report the residuals from comparing actual OOD performance at different epochs to what could be predicted from the corresponding ID performance using this bascline. Refer
3

---

Training set size
Number of classes
Class distribution
Class hierarchy
Expert-labeled
ImageNet
1,281,167
1,000
bals-edl
WordNet
No
iNaturalist
579.184
5,089
Long- tailed
Tree of life
Yes
Table I: Differences between the ImageNet and iNaturalist datasets.
50
50-
samples 5k
samples 5k
40
40
samples 25k
samples 25k
OOD macro
samples 50k
samples -0k
samples 200k
samples 200k
20
samples 150k
20
samples 150k
10-
OOD
10-
5
5-
10
20
30
40
50
60
70
10
20
30
40
50
60
70
Test macro
Test macro
Figure 4: Reducing the number of pre-training images randomly sampled from (left) ImageNet and (right) iNaturalist lowers the robustness linear trends of the fine-tuned models. However, note that using only 25K pre-training samples (green line) still yields significant robustness improvements compared to training from scratch on 129K iWildCam images (dark blue line). We subsample iNaturalist to ensure class balance, by only including 1000 classes with the most number of samples.
to Figure for more details. We find that in the context of iWildCam fine-tume tie-ttnnng, at each epoch, the residuals from architectures of choice concentrate around the y 0 line and exhibit no particular trend. This in turn allows us to vary the number of fine-tuning epochs as hyperparameter, and obtain models covering wide range of test performances for the scatter plots.
3 Experimental Setup
As mentioned earlier, the downstream task of interest is wildlife classification with the iWildCam- WILDS dataset [22 the input is a photo taken by camera trap, and the output is one of 182 different animal species. There are two test sets for evaluation: ID test data consists of images taken by the same camera traps the training set, but on different days from the training and validation (ID) images. In contrast, OOD test data contains images taken by a disjoint set of camera traps from training and validation (ID) images. We include some examples of the geodiversity represented in each test split in Appendix Figure [3] Following [22], we report the macro Fl scores of the trained networks because this metric emphasizes performance on rare species, which is critical the biodiversity monitoring application that the dataset was designed for.
Pre-training datasets. We use ImageNet [T]] and iNaturalist [46] the primary pre-training distri- butions of interest, given their hierarchical structures, complexity, and relevance to the downstream task. The two data sources also differ in many ways (Table D]. hence their pre-trained features make for an informative comparison. We also include experiments with synthetic pre-training data by using Stable Diffusion [35] and the FractalDB-Ik Ik dataset (21). We will elaborate on this in Section 4.5
Network architectures. To obtain data for plotting linear trends, we train a range of standard neural network architectures including ResNet (15), ResNext [48]. DenseNet [20). AlexNet [25) and MobileNet V3 [7]. In our scatter plots, besides varying the architectures, we also vary the number of fine-tuning epochs obtain models with varying Fl scores. Appendix contains further training details, While our focus is on supervised pre-training, we also report some additional results with
CLIP 2 architecture in Section 5
In the subsequent sections, we detail different interventions made to the pre-training distribution to disentangle key properties of interest. We show the resulting linear trends in relation to the trends replicated from previous work [27], which include models trained from scratch on iWildCam (solid blue line) as well as models pre-trained on ImageNet (solid cyan line). For each trend line, we show 95% bootstrap confidence intervals for the linear fit.
Experiment Results
4.1 Effect of Data Quantity
First, we experiment with reducing the pre-training set size. To remove potential confounding effects from long-tailed data distribution, we ensure that the class distribution of our pre-training datasets

---

macro
30
macro
30
classes
17 classes
10
10-
37 classes
85 classes
INaturalist
5
232 classes
5-
Superclass INaturalist
10
20 30
40
50
60
70
10
20
30
40
50
60
70
Test macro
Test macro
Figure 5: Results of changing the label granularity of the pre-training task by combining classes according to some semantics hierarchy to form supersets, for (left) ImageNet and (right) iNaturalist. In general, this intervention lowers model robustness on downstream task. However, extreme reduction of the pre-training label space, e.g. by x in the case of ImageNet, still offers robustness gains compared to training from scratch.
uniform. ImageNet is already class-balanced, but this is not the case for iNaturalist [46). We experiment with a 1000-class subset of iNaturalist using its most frequent classes. We further select images within each class uniformly random that the number of samples is the same across all classes. This results in a class blass-balanced training set of size 150K from iNaturalist. We repeat the same procedure obtain subsets of size 100K, 50K, 25K and SK. A similar subsampling process is done on ImageNet, using the full 1000-class dataset which already has a uniform label distribution.
In Figure 4
observe that reducing the data quantity during pre-training lowers the effective robustness of fine-tuned models. However, it is worth noting at 25K images, pre-training with subsampled ImageNet and iNaturalist data still produces much more robust models compared to training from scratch. This is 6x less data compared to what is used for fine-tuning. As a sanity check, we find that using only 5K samples (i.e., 5 examples per class) during pre-training yields roughly the same level of robustness as training from scratch on iWildCam.
4.2 Effect of Label Granularity
Next, we adapt a question raised previously by Huh et al. [9] to our investigation: how does varying the number of pre- training classes affect downstream robustness? Following [19] we construct supersets of classes ImageNet using the WordNet hierarchy. We use the maximum of the shortest path distance from the root of WordNet to a label to compute the maximum depth of the current label set. We then contract ImageNet label nodes along the shortest path to construct superclasses. Specifically, we investigate depths 2, 4, 5, 6, and 7, which result in class counts of 5, 17, 37, 85, and 232 respectively, in order to provide good coverage across a range of label granularities. Similarly, on iNaturalist, we use the superclass information that comes with the dataset to collapse the label space from 5,089 fine-grained classes to 13 coarse classes.
For ImageNet pre-training, we find that using the full 1000 classes provides the most robustness. However, when the label set size is reduced by four times (i.e., taking 232 superclasses at depth 7), model robustness only decreases slightly. From then on, reducing the label set further to 85 classes (depth 6), and then 37 classes (depth 5), does not deteriorate the linear trend further. Only when we experiment with 17 classes (depth 4) do we find another noticeable reduction in effective robustness. With 5 superclasses as the only pre-training labels (depth 2), pre-trained models still yield significantly more robustness than training from scratch.
On iNaturalist, also observe a similar downward shift in linear trend when we reduce the initial label space to its phylum. Refer to Figure for more details. Overall these findings suggest that using fine-grained labels during pre-training is better for learning representations that are robust to distributions shifts in downstream tasks. But even if only coarse labels are available, pre-training with enough data still has significant robustness benefits offer.
4.3 Effect of Label Semanties
The number of pre-training classes seems to have an impact on downstream robustness, but does it matter what kind of classes models are pre-trained on? We next investigate whether using classes whose semantics are more aligned with the downstream task would improve robustness.
5

---

50
Animals Objects
Figure 6: Varying the semantic category classes included in the pre-training data yields similar robustness linear trends, with pre-training only on "animal" classes exhibiting slightly higher F1 scores than pre-training only on "object" classes. Even though the downstream task is animal classifi- cation, models pre-trained only on "object" classes are still much more robust than models that do not undergo any pre-training.
OOD Test macro
20 30
40
50
60
70
ID Test macro F1
To do so, separately pre-train models ImageNet classes that subsets of the "'bjectt" and "animal" WordNet synsets. This yields 2 broad categories that are similar in total sample size, each having around 600K images. In Figure 6 find that models trained "animal" classes (yellow line) exhibit slightly higher Fl scores, but roughly the same effective robustness as models trained on "object" classes (green line). This is surprising given that the fine-tuning distribution, iWildCam. contains only images of animals in the wild, which are semantically more similar to the animal classes of ImageNet. is worth noting that models pre-trained on "object" classes are also much more robust than models trained from scratch (blue line).
One potential confounder to this experiment setup is that some images from "object" classes also contain animals (i.e.. co-occurrences that are not accounted for by ImageNet labels). To estimate the extent of this problem, we use TensorFlow's ImageNet2012 multilabel ImageNet validation data with multi-class labels reviewed by experts. We find that 1.1% of the data have labels from both "animal" and "object" classes present in the same image, suggesting that the label co-occurrence issue only impacts small fraction of training data. Consequently, we posit that training on diverse set of classes in general helps the model pick up on useful invariances that in turn lead to similar downstream robustness. We explore this hypothesis further in Section 4.5 with synthetic training data.
[1], containing 20k
4.4 Effect of Image Diversity
Besides labels, another source of diversity comes from the training images themselves. We experiment with two different notions of image diversity (i) Label diversity, and (ii) Per-class image diversity.
4.4.1 More Data Per Class vs. More Classes of Data
A natural question arises when designing dataset with fixed data budget (or labeling cost): should we collect more data from existing categories or more categories of data? To address this question, keep the total number of images fixed while varying the number of classes of ImageNet we use for pre-training. For example, if we have a budget of 60K images and 100 randomly selected ImageNet classes, sample 600 images uniformly random from each of these classes (Figure 7p. Here, we find that in the context of the iWildCam distribution shift, there is no difference on downstream robustness between having more data per class having more classes, as long as the total number of images is constant. This observation also holds at larger data quantity scale (300K images, see Appendix Figure 18 This result demonstrates the dominant effect of data quantity over other aspects of the pre- treiiiaiingg distribution (e.g., label set size).
4.4.2 Image Diversity Within Each Class
Another way of modulating dataset diversity is by changing per-class image diversity. For example, given "dog" class, dataset which only contains images of one dog breed could be seen less diverse than a dataset which has examples of several breeds. In order to construct a controlled experiment, we use quantitative heuristic for the diversity of each class: we fix certain superclasses (using the WordNet hierarchy) as the training labels and vary the number of corresponding subclasses where the images are taken from. For iNaturalist More diversity means more subclasses chosen per superclass.
do the same with the tree of life structure.
For the ImageNet distribution that is built on the WordNet hierarchy, we construct a subset following BREEDS [39]. The two main ways that BREEDS recalibrates the WordNet hierarchy fit our goals for image diversity: (i) selected nodes convey some visual information, and (ii) nodes of similar

---

50
Figure 7: We vary the number of classes randomly selected from the original 1000 ImageNet classes and adjust the number of images per class corre- spondingly, such that total image quantity is 60K. We observe that having 4x more classes, or 4x more images per class, induces the same level of robustness in fine-tuned models. Experiments with 300K data regime can be found in Appendix Figure
OOD Test macro
# of classes = 50
# of classes 100
of
18
20 30
40
50
60
70
ID Test macro F1
50
50
p = 0.33
p 0.33
p 0.67
p 0.67
Test macro
30
p
30
p
20
20
10
30
40
50
70
10
40 50
70
ID Test macro F1
ID Test macro F1
Figure 8: We fix the total amount of pre-training data and the label space, while reducing the number of subclasses that constitute each superclass label in (left) ImageNet and (right) Naturalist. Smaller (diversity ratio) means proportionally fewer subclasses per superclass. We find that reducing per-class diversity by up to 3x has no effect on the robustness of downstream models.
specificity share the same distance from the root (e.g., "dog" and "cat" are now both the same depth even if the "dog" subtree is much larger). With this new hierarchy, we obtain 16 superclasses, each encompassing 12 subclasses (i.e., original ImageNet classes). The full list can be found in Appendix C.I] We vary image diversity by changing the number of subclasses per superclass: 4, 8 or 12 Subclasses--oorespppnne correspondinggggg to the diversity ratio 0.33, p 0.67, 1.0 in the left panel of Figure 88 To prevent data quantity from being a confounding variable, we subsample images from each chosen subclass accordingly (e.g., if reduce number of subclasses per superclass by 3x then we sample 3x more images from each subclass).
For iNaturalist, we fix the total number of images 80K and apply the same procedure described above to select a fraction of subclasses (see diversity ratio values in the right panel of Figure 3), for each of the following superclasses: "Plantae", "Insecta", "Mammalia", "Fungi", "Aves", "Reptilia", and "Amphibia." We choose this set of superclasses so we could have a uniform distribution of images per class while maintaining the same number of images as ImageNet experiment. For more details, see Appendix C.I. As seen in Figure & for both ImageNet and iNaturalist, the resulting linear trends are highly similar regardless of the diversity ratio p. or the number of subclasses per superclass. We conclude that in this case, per-class image diversity does not have a significant impact on downstream robustness. Note that this does not hold in the extreme setting, e.g. repeating the same image to produce a dataset.
4.5 Pre-training with Differentt Data Sources
Moving beyond interventions within each data distribution, compare fine-tuning robustness behaviors across different data sources.
Compared to ImageNet, iNaturalist exhibits different characteristics on multiple axes (see Table I We expect that pre-training on the diverse, domain-speciff species which have been verified by nature enthusiasts in iNaturalist will provide boost on robustness for the downstream animal- in-the-wild classification task, compared to training on general web-curated classes in ImageNet. However, in Figure 10 we find that iNaturalist behaves similarly to ImageNet as a pre-training data source, Even when we subsample iNauuralist to follow the ImageNet class distribution (refer to Section 4. for its construction), observe similar level of effective robustness compared to
7

---

Figure 9: Each grid in order shows random examples from the ImageNet ILSVRC 2012 challenge train set [37[[]] [1]], the iNaturalist 2017 challenge train set [46], the FractalDB 1k synthetic train 21 and a 1000-class ImageNet-styye synthetic dataset generated using Stable Diffusion [3]].
iNaturalist
FractalDB 1k
Figure 10: Pre-training on noisy, long-tailed dis- tribution of natural images like iNaturalist (red line) does not change the robustness on down- stream task, compared to pre-training on clean, dataset like ImageNet (cyan line). Pre-training on the same amount synthetic frac- tal data (FractalDB-Ik Ik) yields much lower robust- ness (green line), but still has some benefits com- pared to training from scratch (dark blue line).
0OD Test macro
20
30
40
50
60
70
ID Test macro F1
the equal-sized 150K ImageNet subset (Figure 1T We hypothesize that when certain level of "diversity" is reached with the training images and labels, there is negligible robustness gain to be made even if inerease the alignment between the pre-training and fine-tuning data domains.
4.5.1 Synthetic Data Sources
To push our diversity hypothesis to the limit, we also pre-train the same set of architectures on FractalDB-Ik dataset [21. which has similar class distribution to ImageNet but only contains synthetic fractal images. Pre-training on FractalDB- Ik has been shown to surpass the accuracy of pre-training ImageNet/Places 211. For the task of iWildCam WILDS, however, it is noticeably less effective at improving downstream robustness compared to natural image data (Figure I0P. However, pre-training with fractal images still offers more robustness than training from scratch.
Can we generate better synthetic data for pre-training than FractalDB- 1k? We experiment with Stable Diffusion (35), popular diffusion model which generates high-qualit, images from natural language prompts, to generate natural-looking images following the ImageNet class distribution. We use 80 diverse prompts per ImageNet class from (32 to generate 150K ImageNet-like dataset. Examples from this synthetic dataset can be seen in Figure O We find that pre-training on this dataset yields similar robust generalization behaviors as using the same quantity of ImageNet and iNaturalist images (Figure TT). However, at larger scale of IM images, the robustness benefits of pre- traininingg with synthetic data begins to saturate and slightly lag behind iNaturalist and ImageNet. See Appendix Figure 19 for more details.
Overall our findings demonstrate that while nuances in image semantics during pre-training are not important for fine-tuning robustness (e.g., Animals versus Objects classes, iNaturalist versus ImageNet), it is still beneficial to match the general characteristics of the downstream data (e.g.. "natural- lookkngg images).
5 Self-supervised Pre-training
Previous experiments revolve around the supervised learning settings. However, it is increasingly common to pre-train on self-supervise tasks using web-crawled corpora, which has been shown to significantly improve robustness to distribution shifts [32]. Our preliminary experiments with pre-trained CLIP models 332 on iWildCam show that the resulting and OOD performances still lie on the ImageNet pre-training linear trend (i.e., cyan line), despite the self-supervised training mechanism and the much larger training dataset of CLIP. Varying CLIP's data sources only move the Fl scores along the same line (Figure
8

---

INaturalist 150k
ImageNet 150k
OOD Test macro
Diffusian 150k
Figure 11: Similar to results in Figure TO) with budget of 150K images, pre-training on FractalDB- 1k is significantly less effective than iNaturalist or ImageNet. However, we find that generating natural-looking data with Stable Diffusion can help close this gap. All datasets are subsampled to be bass-balanced, each having 1000 classes.
FractaIDB-1k 150k
20
30
40
50
60
70
ID Test macro F1
Figure 12: Results from fine-tuming CLIP [32 pre- trained models on iWildCam-Wil -IILD The mod- els we use include CLIP with ResNet-50 image encoder trained on YFCC- [45] and LAIIO- 15M [40] respectively, as well as all original CLIP models released by OpenAl, including 3 with ViT [12] backbone, trained on dataset of size 400M. All of these models lie on the same linear trend as that of ImageNet pre-training, demonstrat- ing the consistency this trend across many pre- training dataset size scales and training algorithms.
40
00D Test macro
30
20
10
OpenAl CLIP
LAION-15M (RN50)
5
YFCC-15M 15M (RN50)
10
20
30
40
50
60
70
ID Test macro
We also repeat the experiments with varying data quantity (Section 4.1 in the context of CLIP's image-text pre-traae pre-training data. Refer Appendix [E more details. We leave an extensive evaluation of how "diversity" of the pre-training distribution should be defined and measured differently on these open-vocabulary web datasets future work.
Conclusion & Discussion
In this work, we find that many factors during pre-training such as label semantics and image diversity, do not significantly alter the effective robustness of models fine-tuned on WilldCa-WWILDD. The more influential factors for downstream robust generalization are the quantity of the pre-training data and the granularity of the label set. Through experiments with Stable Diffusion, we also demonstrate the potential of synthetic natural-looking images as a way to increase the effectiveness of the pre-training distribution along these two ablation axes.
We can think about pre-training dataset construction terms of an explore vs. exploit trade-off. Exploration, such finding new data sources, is often time consuming, while exploiting, or collecting as much data as possible from an existing source, can sometimes experiments suggest that good approach to building pre-training dataset for robust generalization is to find a few data sources that exhibit robustness characteristics on a downstream task (e.g., Stable Diffusion data), and then collect as many data points from these sources possible.
significantly casier. Our
important to note that we are studying a different model behavior than Huh al. 19 Some interventions can reduce average performance while maintaining effective robustness (e.g., label granularity in Figure 5) and vice-versa (e.g., architecture modifications). Thus, certain choices during pre-training dataset design depend fundamentally on the goals of the dataset. For example, whether we want to achieve consistent performance across many settings (i.c., robustness) or optimize for specific application changes the applicability of our results.
very good performance
An important open question determining what characteristic of the WildCam-WILLD task leads to the difference in linear trend between pre-training and training from scratch. Some other datasets (e.g., EMoW-WILDS, see pinpoint distribution shifts where pre-training can provide significant boost in effective robustness. Finding a unifying property among such datasets would allow for better interpretation of our current results. In Appendix G we look into distribution shift settings constructed from the DomainNet benchmark [30], and observe that pre- pre-training and training from scratch only produce different linear trends for certain pairs of domains and not others. Refer to this Appendix for further discussion.
do not exhibit this behavior after fine-tuning, so it important to

---

7 Acknowledgements
This work is supported in part by Open Philanthropy, the Allen Institute for AI, and NSF grants DMS- 2134012 and CCF-2019844 as a part of NSF Institute for Foundations of Machine Learning (IFML). Ali Farhadi acknowledges funding from the NSF awards IIS 1652052, IIS 17303166, DARPA N66001- 19-2-4031, DARPA 1911NF-15-11054 and gifts from Allen Institute for Artificial Intelligence, Google, and Apple.
References
[1] A. Andreassen, Y. Bahri, B. Neyshabur, and R. Roelofs. The evolution of out-of-distribttion robustness throughout fine-tuning, arXiv preprint arXiv:2106.1580 2021.
[2] S. Beery, E. Cole, and A. Gjoka. The iwildcam 2020 competition dataset. arXiv preprint arXiv:2004. 10340, 2020.
[3] B. Biggio and Roli. Wild patterns: Ten years after the rise of adversarial machine learning. 2018. 2018.
Pattern Recognition, 84:317-331,
[4] B. Biggio, I. Corona, D. Maiorea, B. Nelson, N. Šmdić. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery databases, pages 387-402. Springer, 2013.
[5] L.-C. Chen, G. Papandreou, Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IFFE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[7] G. Christie, N. Fendley, J. Wilson, and R. Mukherjee. Functional map of the world. In CVPR, 2018.
[8] A. Conneau and D. Kicla. Senteval: An evaluation toolkit for universal sentence representations. arXiv preprint arXiv: 180300549, 2018.
[9] J. Dai, Y. Li, K. He, and J. Sun. R-fen: Object detection via baaeed fully convolutional networks, Advances neural information processing systems, 29, 2016.
[10] De Fauw, R. Ledsam, B. Romera-Paredes, S. Nikolov, N. Tomasev, S. Blackwell, H. Askham, X. Glorot, B. O'Donoghue, D. Visentin, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, 24(9): 1342-1353335 2018
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 Icee, 2009.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[13] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). arXiv preprint 22050220500199 2022.
[14] Y. Guo, H. Shi, A. Kumar, K. Grauman, T. Rosing, and R. Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1805-4814 2019.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
10

---

[16] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969. 2017.
[17] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang. Y. Zhu, R. Pang, V. Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314-1324, 2019.
[18] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, Guadarrama, et al. Speed/aceuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 73 7310-7311, 2017.
[19] M. Huh, P. Agrawal, and A. A. Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv: ww...6008...00 08. 2016.
[20] F. landola, M. Moskewicz, S. Karayev, R. Girshick, T. Darrell, and K. Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv: 1869, 2014.
[21] H. Kataoka, K. Okayasu, A. Matsumoto, E. Yamagata, Yamada, N. Inoue, A. Nakamura, and Y. Satoh. Pre-training without natural images. International Journal of Computer Vision (IJCV), 2022.
[22] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of wn-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[23] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung. S. Gelly, and N. Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491-507 Springer, 2020.
[24] Kornblith, Shlens, and Q. Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2661-2671, 2019.
[25] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105,, 2012.
[26] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.51001, 2017.
[27] P. Miller, R. Taori, A. Raghunathan, Sagawa, P. W. Koh, V. Shankar, P. Liang, Carmon, and L. Schmidt. Accuracy on the line: on the strong correlation between outto--distribution and in-distribution generalization. International Conference on Machine Learning, pages 7721-7735. PMLR, 2021.
(28) R. Mormont, P. Geurts, and R. Marée. Comparison of deep transfer learning strategies for digital pathology. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 2262-2271, 2018
[29] T. Nguyen, G. Ilharco, M. Wortsman, Oh, and L. Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. arXiv preprint arXiv:2208.0553 2022.
[30] X. Peng. Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[31] J. Quinonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning. Mit Press, 2008
[32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-878 8763. PMLR, 2021.
11

---

[33] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[34] Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[35] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[36] A. Rosenfeld, R. Zemel, and J. K. Tsotsos. The elephant the room. arXiv preprint arXiv: 03305, 2018.
[37] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, Ma, Z. Huang, A. Karpathy. A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252. 2015.
[38] H. Salman, A. Ilyas, Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 2020.
[39] S. Santurkar, D. Tsipras, and A. Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008. 04859, 2020.
[40] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, Mullis, A. Katta, T. Coombes,
J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
[41] V. Shankar*, R. Roelofs". Mania, Fang, B. Recht, and Schmidt. Evaluating machine accuracy on imagenet. ICML, 2020. http://prroceedings mlr, html
http:///gn..nn.pprresssss////aaa presss//119/hhamaa2000
[42] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[43] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:/312.6199 2013.
[44] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-115 18599, 2020.
[45] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Y0cm The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[46] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8769 8769-8778, 8778, 2018.
[47] H. Wang. S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506- 105 18518.7 2019.
[48] S. Xie, R. Girshick, Dollár, 7. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1492-1500, 2017.
[49] K. You, Z. Kou, M. Long, and J. Wang. Co-tuning for transfer learning. Advances in Neural Information Processing Systems, 33:17236-17248 17246, 2020.
12


Reviews:
[{"review_id": "gV4GOa32pb", "reviewer": "Reviewer_XBK9", "summary": "This paper investigates the role of pre-training data diversity on fine tuning robustness. They vary various factors like label space, label semantics, image diversity, data domains, and data quantity of the pre-training distribution to investigate how these factors impact the robustness of the models. Some interesting insights include similar label semantics doesn't necessarily improve the robustness of the model and increasing per-class examples doesn't necessarily improve the robustness of the model.", "questions": "- Can these results generalize to other downstream task as well? In this work only iWildCam-WILDS dataset is incorporated into results, it would be interesting to see if these results generalize to other datasets and datasets from different domains. ", "limitations": "addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Provides insights on pre-training models like how various factors like data diversity, label space, label semantics etc affect the robustness of the models. Might be helpful for ML practitioners trying to decide what kind of data is best suited for pre-training \n- Experiments are thorough and clean. Paper is also easy to read highlighting main results.\n\n", "weaknesses": "- Major section of the work has been focused on supervised pre-training which is becoming less and less common with the advent of self-supervised learning methods. It would have been interesting to look at these of pre-training strategies in much more depth.\n- Only one downstream task considered in the experiments (iWildCam-WILDS). Hard to quantify if these results generalize to other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the effort they have put into reviewing our paper.\n\n**“It would have been interesting to look at self-supervised learning methods in much more depth.”** This is a good direction for future work. The focus of our current work is on supervised pre-training. Per the reviewer’s feedback, we have included experiments with SimCLR. In Figure 2 (right) of the rebuttal PDF, we include a comparison of a ViT-B/32 model pre-trained with SimCLR on ImageNet and the same architecture pre-trained on ImageNet in a supervised fashion. We see that models trained on the same dataset exhibit similar robustness trends despite the differing pre-training strategies. We note that certain properties of the pre-training distribution that we examined are harder to control/not applicable in the self-supervised learning regime (e.g. label granularity), which is why it is not the main focus of the paper.\n\n**“Only one downstream task considered in the experiments (iWildCam-WILDS). Do these results generalize to other datasets?”** This is a notable limitation which we acknowledge in the Conclusion section. We do study some of the properties of the pre-training distribution in the context of DomainNet (see Appendix E) and show that our main findings on the impact of label granularity and data quantity hold. Overall, characterizing distribution shifts where pre-training on non-web-scale datasets can provide a significant boost in effective robustness compared to training from scratch is an important direction for future work. The effect of varying pre-training distributions on downstream effective robustness has been studied at larger scales across a variety of distribution shifts (e.g. [1]), but at smaller scales the only shift that consistently sees substantial robustness benefits from small-scale pre-training is iWildCam-WILDS [2], making it a useful testbed.\n\n[1] Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP. Nguyen et al., 2022.\n\n[2] Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization. Miller et al., 2021.\n"}, {"review_id": "kVk7a2wRvB", "reviewer": "Reviewer_6SSm", "summary": "In this paper, the authors investigate the influence  of pre-training data on the robustness of fine-tuning. The authors design several experiments by following a common pre-training, fine-tuning and evaluation pipeline. They  found that the quantity of the pre-training\ndata and the granularity of the label set  are two most influential factors on the robustness of downstream fine-tuning. The  authors further leverage synthetic data from Stable Diffusion to increase the effectiveness of the pre-training distribution along these two ablation axes.", "questions": "1. In figure 3, why the fine-tuning epoch is 12?\n\n2. Would the number of fine-tuning epochs affect the  robustness of the pre-trained model? The authors found that by only using 5K samples for pre-training offers no robustness gain, this raises the question that if longer fine-tuning will also reduce robustness. ", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is  well-written and the motivation is clear.\n2. The authors considers many factors that will possibly  affect the fine-tuning robustness including data quantity, label granularity, label  semantic, etc.\n3. The conclusions of the paper are clear which is useful for follow-up research. ", "weaknesses": "1. The  authors mainly focus on the impact of the  pre-training data on the fine-tuning robustness, it is not clear if different fine-tuning methods (only fine-tune the last layer, etc) would change the conclusions of the paper.\n2. The authors considers out-of-distribution generalization in this paper, it is unclear whether the conclusions also apply to other concepts of robustness, such as adversarial robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback!\n\n**“It is not clear if different fine-tuning methods (only fine-tune the last layer, etc) would change the conclusions of the paper.”** This would be a useful direction for future investigation. Our paper includes experiments with linear probes for CLIPs pre-trained on different data sources (Figure 12). There we find that the fine-tuned models lie on the same linear trend as models pre-trained on ImageNet and fine-tuned end-to-end. We will include experiments with different fine-tuning methods in the subsequent version of our paper. We expect this to follow similar linear trends as these kinds of training interventions have been studied previously in [1].\n\n**“The authors consider out-of-distribution generalization in this paper, it is unclear whether the conclusions also apply to other concepts of robustness, such as adversarial robustness.”** This is an interesting point. We are primarily interested in the study of natural distribution shifts as current paradigms for pre-training seek to have a very general backbone that encompasses a wide range of data, in order to be robust to many downstream natural distribution shifts (e.g. CLIP). We believe that having a better understanding of how to choose pre-training distributions to be robust to these kinds of shifts would be highly relevant. We agree that studying other robustness concepts is an interesting avenue for future work, but is not a focus of our current work, especially since adversarial robustness itself can be defined in many different ways in existing literature.\n\n**“In figure 3, why the fine-tuning epoch is 12?”** This choice follows from the experiment setup in previous work [1]. In Figure 3, using linear trends obtained from [1], we also study the impact of the number of fine-tuning epochs on how well the resulting model’s performance fits the linear trend, and find that varying the number of epochs does not change the linear trend significantly. We also include new experiments with longer fine-tuning (see Figure 1 (left) of the rebuttal PDF).\n\n**“The authors found that by only using 5K samples for pre-training offers no robustness gain, this raises the question that if longer fine-tuning will also reduce robustness.”** Per your comment, we have added new experiments fine-tuning models pre-trained on different data quantities (5K and 25K) for double the number of epochs. We observe that longer fine-tuning does not change the resulting trend lines. Refer to Figure 1 (left) of our rebuttal PDF for more details.\n\n[1] Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization. Miller et al., 2021."}, {"review_id": "MgVWxBxSdi", "reviewer": "Reviewer_ev5Y", "summary": "This paper delves into the impact of pre-training data construction on fine-tuning robustness, encompassing aspects such as dataset size, class granularity, in- and out-class diversity, class similarity, and the use of synthetic data for pre-training. The evaluation metric employed is the in- and out-of-distribution testing performance, assumed to follow a linear trend. The study investigates changes in the slope of this trend to assess the influence of the mentioned factors. The findings suggest that pre-training data quantity and label granularity significantly affect fine-tuning robustness.\n", "questions": "In addition to the general limitations mentioned in the ''Paper Weakness'' part, I have several additional questions:\n\n1. Incorporating transformer networks into the experiments would be beneficial, considering their increasing popularity in vision tasks.\n\n2. Addressing the influence of networks in the experiments, particularly in Sec. 4.1 \"Effect of Data Quantity,\" is crucial. Larger models might suffer from reduced training dataset sizes, while smaller models could perform well under such circumstances. It is essential to mitigate this influence, as the plotted performance directly relates to different models.\n\n3. The absence of a discussion on the influence of long-tailed data distribution is noteworthy. Understanding whether imbalanced pre-training datasets affect fine-tuning robustness is crucial. Treating data distribution merely as a confounding effect might not be sufficient.\n\n4. In some cases, such as Figure 4 and 11, the linear trend is not readily apparent, and the scattered data points seem to follow more irregular patterns. Further clarification on these trends and their implications would be helpful.", "limitations": "The authors have addressed some limitations of their work, and there are additional suggestions for improvement in the 'Paper Weakness'' part and \"Questions\" part.\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper presents a comprehensive exploration of the relationship between pre-training datasets and fine-tuning robustness. The novel perspective on dataset construction offers valuable insights into deep learning models.", "weaknesses": "A potential limitation lies in the reliance on a single metric to measure effective robustness, which might not provide a fully comprehensive evaluation of the connection between pre-training datasets and fine-tuning robustness.\n\nThe figures' lines and points lack sufficient definition, which makes it somewhat challenging to grasp their full meaning. Some lines appear to be extrapolated, raising concerns about their validity beyond the tested environments, as observed in Figure 4, 5, 6, 7, 8, 10, and 11.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for providing valuable suggestions on how we can improve our work.\n\n**“The reliance on a single metric to measure effective robustness, which might not provide a fully comprehensive evaluation of the connection between pre-training datasets and fine-tuning robustness.”** Effective robustness trends have been shown to hold for a variety of interventions, including training method, hyperparameters and dataset size [2]. Besides, robustness could be defined in many different ways in existing literature. Given the various interventions we could perform on the pre-training distribution, we chose to focus on robustness to natural distribution shifts, which has become increasingly important in the context of large-scale pre-training.\n\n**“The figures' lines and points lack sufficient definition, which makes it somewhat challenging to grasp their full meaning.”** Thank you for the useful feedback. We provide background on the effective robustness framework as well as what the points represent in Section 2 (Background). We will improve the clarity as well as the context with respect to related work in subsequent revisions. The reviewer could also refer to previous work [1, 2] for more details.\n\n**“Some lines appear to be extrapolated, raising concerns about their validity beyond the tested environments, as observed in Figure 4, 5, 6, 7, 8, 10, and 11.”** Thank you for the important observation. Concerns around extrapolation are the reasons why we provide error bars for the reported linear trends in the form of bootstrap confidence intervals. Besides, across the range of performance currently attainable on the downstream task (iWildCam-WILDS), we find that the F1 scores resulting from our interventions overlap significantly, allowing us to fairly compare the robustness properties across different pre-training distributions that yield similar performance range. \n\n**“Incorporating transformer networks into the experiments would be beneficial, considering their increasing popularity in vision tasks.”** Per your comment, we report new results with ViTs in Figure 2 of our rebuttal PDF, specifically ViT-B/32, pre-trained in both supervised and self-supervised (SimCLR) manners. We observe that our data quantity result holds for the ViT-B/32 architecture. We will expand our analysis of ViT with more architectures in the subsequent versions of the paper.\n\n**“Larger models might suffer from reduced training dataset sizes, while smaller models could perform well under such circumstances. It is essential to mitigate this influence, as the plotted performance directly relates to different models.”** This is an interesting point to study. In Figure 1 (right) of the rebuttal PDF, we plot the residuals to the overall trend line of pre-training on a fixed data quantity (shown in Figure 4 of the main paper), for each architecture size. We find that the residuals do not exhibit any particular pattern and remain relatively consistent across different model sizes.\n\n**“The absence of a discussion on the influence of long-tailed data distribution is noteworthy. Understanding whether imbalanced pre-training datasets affect fine-tuning robustness is crucial.”**  We do include a discussion of pre-training on long-tailed data in our paper. Our experiments involve pre-training on two versions of iNaturalist: the raw, long-tailed data (Figure 10) and the 1000-class, class-balanced subset of iNaturalist using its most frequent classes (Figure 11). In both cases, we find that iNaturalist behaves similarly to ImageNet in terms of the robustness properties. We discuss these findings in Lines 239 - 248 of our paper.\n\n**“In some cases, such as Figure 4 and 11, the linear trend is not readily apparent, and the scattered data points seem to follow more irregular patterns.”** For certain experiments (e.g. pre-training with 50K or 100K images, Figure 4), our interventions make little difference on the downstream robustness. Consequently, the linear trends end up overlapping. Per the reviewer’s concern, we report the linear fit of each trend line in Figure 4 and Figure 11 in the following table. We note that the $R^2$ is high for all of the linear trends. The “irregular data pattern” effect is largely a result of the intervention not producing a large change in linear fit, making our data points seem like a large cluster. We will include separate plots for each subsample experiment as well as add this $R^2$ analysis to the appendix.\n\n| Pre-Training Dataset | Samples | Coefficient of Determination ($R^2$) |\n| ------------------- | ------- | ---------------------------------- |\n| Imagenet            | 5k      | 0.634                            |\n| ImageNet            | 25k     | 0.799                            |\n| ImageNet            | 50k     | 0.874                            |\n| ImageNet            | 100k    | 0.769                            |\n| ImageNet            | 150k    | 0.862                            |\n| iNaturalist         | 5k      | 0.689                            |\n| iNaturalist         | 25k     | 0.741                            |\n| iNaturalist         | 50k     | 0.873                            |\n| iNaturalist         | 100k    | 0.824                            |\n| iNaturalist         | 150k    | 0.908                            |\n| iNaturalist         | 150k    | 0.908                            |\n| ImageNet            | 150k    | 0.862                            |\n| Diffusion           | 150k    | 0.906                            |\n| FractalDB-1k        | 150k    | 0.565                            |\n\n\n[1] Measuring robustness to natural distribution shifts in image classification. Taori et al., 2020.\n\n[2] Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization. Miller et al., 2021."}, {"review_id": "XC84ucwF56", "reviewer": "Reviewer_t8Yd", "summary": "The paper introduces an empirical study for visual pre-training. By focusing on the pre-training, the authors conduct wide-range experiments through (i), data quantity, (ii) label granularity, (iii) label semantics, (iv) image diversity, (v) data sources. The empirical study considers how to use pre-training datasets (both real and synthetic datasets), fine-tuning (data distribution), and test tasks (image classification, out-of-distribution detection). In the experimental section, the study disappears several findings for visual pre-training.", "questions": "- See above-mentioned questions", "limitations": "There are no negative limitations and societal impacts.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The novelty of the project is experimental findings with a wide-range of visual pre-training. The paper reveals five items in effect of data quantity (Section 4.1), label granularity (Section 4.2), label semantics (Section 4.3), image diversity (Section 4.4), and pre-training with different data sources (Section 4.5).", "weaknesses": "- How about implementing vision transformer (ViT) architecture? According to the experimental setup in Section 3, almost all of the network architectures are based on CNN architecture. Although some additional results are reported CLIP, one of the current visual architectures is undoubtedly ViT architecture. Since many architectures are implemented in the experiments, it doesn't seem that there is a lack of computing resources for the experimental conduction.\n\n- The reviewer would touch the results shown in Figure 4. Does the graph show the highest score in the upper right corner? The claimed point with 25k pre-training samples (green line) is better than the baseline with 129k iWildCam images, but still wouldn't it be better to have more data such as 100k and 150k in terms of robustness? With this results, can we conclude tha \"we do not need a lot of pre-training data to see significant robustness gains\" (l.46-47)? \n\n- Related to Section 4.5.1 and Figure 10, the PixMix framework [Hendrycks+, CVPR22] with the combination of real and synthetic images improves a robustness score. Therefore, for robustness performance, the present paper should implement PixMix framework rather than that of single usage of fractal images.\n\n[Hendrycks+, CVPR22] Dan Hendrycks et al. \"PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures\" in CVPR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback.\n\n**“How about implementing vision transformer (ViT) architecture?”** We understand that ViTs have gained popularity in recent years. We include results with ViTs in Figure 2 of our rebuttal PDF, specifically ViT-B/32, pre-trained in both supervised and self-supervised (SimCLR) manners. We show that our data quantity result holds for the ViT-B/32 architecture, and that supervised and self-supervised pre-training yields the same linear trends. Due to time constraints, we could not incorporate other ViT architectures for our current suite of experiments, however we will include the ViT-B/32, B/16, and L/14 architectures for subsequent versions of the paper.\n\n**“Does the graph show the highest score in the upper right corner?... With this results, can we conclude that ‘we do not need a lot of pre-training data to see significant robustness gains’ (l.46-47)?”** To clarify your question, the highest score is in the upper right hand corner, and we plot in-distribution and out-of-distribution performance of the same model. Perfect robustness is having the same performance on both distributions (the dotted black line in our plots). However note that a low-accuracy random classifier would also have perfect robustness, so there’s some nuance to this statement. \n\nThe statement in Lines 46-47 does not contradict our findings regarding the impact of pre-training data quantity (Section 4.1). Of course pre-training on significantly more data will lead to further robustness improvement on downstream tasks (e.g. 150K versus 25K trend lines, see Figure 4). However, in Lines 46-47, we want to highlight that even with access to limited pre-training data (e.g. 25K), it is still beneficial to do pre-training, rather than training from scratch on the target task.\n\n**“For robustness performance, the present paper should implement PixMix framework rather than that of single usage of fractal images.”** Thank you for the pointer to this paper. The goal of our experiments with data sources (Section 4.5) is to disentangle the impact of the type of images (e.g. synthetic vs real), as well as the varying characteristics of the data distributions (e.g. long-tailed vs class-balanced). We acknowledge that there exist strategies to better combine signals from both real and synthetic sources to improve robustness. However, exploring data mixing strategies is beyond the scope of our investigation."}]

Author rebuttal:
We thank the reviewers for their thorough, insightful comments and have made revisions based on their feedback. We are glad they found the work well motivated, novel, and the contributions to be of value to the community. Here we include new results and plots to respond to some of the concerns by reviewers, see the attached PDF. We are happy to engage in further discussion and interested in any additional feedback the reviewers may have.

Meta-review: This paper presents a comprehensive empirical study examining the impact of pre-training data diversity on fine-tuning robustness. The conducted experiments encompass a wide array of factors, including data quantity, label granularity, label semantics, image diversity, and data sources. The resulting findings offer valuable insights for practitioners seeking to optimize pre-training models. The reviewers also raised concerns on its clarity and insufficient downstream task evaluation.
Overall, the meta-reviewer deems the contribution of this paper to be substantial and recommends it for publication. The authors are encouraged to incorporate the feedback provided by the reviewers when preparing the camera-ready version.
Decision: Accept (spotlight)