Distributionally Robust Classification on a Data Budget
Authors: Benjamin Feuer
Ameya Joshi
Minh Pham
Chinmay Hegde
Venue: Accepted by TMLR ()

Abstract: Real world uses of deep learning require predictable model behavior under distribution shifts. Models such as CLIP show emergent natural distributional robustness comparable to humans, but may require hundreds of millions of training samples. Can we train robust learners in a domain where data is limited? To rigorously address this question, we introduce JANuS (Joint Annotations and Names Set), a collection of four new training datasets with images, labels, and corresponding captions, and perform a series of carefully controlled investigations of factors contributing to robustness in image classification, then compare those results to findings derived from a large-scale meta-analysis. Using this approach, we show that standard ResNet-50 trained with the cross-entropy loss on 2.4 million image samples can attain comparable robustness to a CLIP ResNet-50 trained on 400 million samples. To our knowledge, this is the first result showing (near) state-of-the-art distributional robustness on limited data budgets.

Published in Transactions on Machine Learning Research (08/2023)
Distributionally Robust Classification on a Data Budget
Benjamin Feuer
b9996 nyu.eda
Depariment of Computer Science and Engineering New York University
370 Jay St., 11 Fl., Brooklyn, NY, 11201
Ameya Joshi
ameya.joshi@nyu.co nyu
Department of Electrical and Computer Engineering New York University
Minh Pham
mp @ nyu.
Department of Computer Science and Engineering New York University
Chinmay Hegde
chinmay.h nyu.edu
Depariment of Computer Science and Engineering New York University
Reviewed on httpp//////aaarreeee.///////eeaa
Abstract
Real world uses of deep learning require predictable model behavior under distribution shifts. Mod- els such as CLIP show emergent natural distributional robustness comparable to humans, but may require hundreds of millions training samples. Can we train robust learners a domain where data limited? To rigorously address this question, we introduce JANuS (Joint Annotations and Names Set), collection of four training datasets with images, labels, and corresponding captions, and perform series of carefully controlled investigations of factors contributing robustness in im- age classification, then compare those results to findings derived from large-scale meta-ane anta--naaysis. Using this approach, we show that standard ResNet-50 trained with the cross-entropy loss on 2.4 million image samples can attain comparable robustness to a CLIP ResNet-50 trained on 400 mil- lion samples. our knowledge, this the first result showing (near) state-of-the-ar distributional robustness on limited data budgets.
1 Introduction
1.1 Motivation
A natural distribution shift is defined as evaluation data which differs from the data on which model was trained due to natural factors. Real world uses of deep image classifiers require predictable model behavior under such shifts. Un- fortunately, several "standard" image classification models perform significantly worse under natural shifts (Hendrycks & Dietterich, 2019 9; Miller et al., 2021), in contrast with human vision (Recht et al., 2019).
Vision-Language (VL) models such as CLIP, introduced in Radford et al. (2021), have shown emergent natural distri- butional robustness comparable humans across a wide range of shifts of ImageNet, at the cost of base accuracy. Jia et al. (2021) showed CLIP-like models can be carefully fine-tuned to be robust as well as achieve high base accuracy, However, such models require massive amounts of data for training, and in some cases, orders of magnitude more samples than standard supervised models (Pham et al., 2021).
These results raise challenging questions: are data scaling laws work for robust computer vision, similar to those discovered in NLP tasks? Does robustness only emerge when models are trained on massive datasets? And is vision- language pre-training necessary for robustness? Radford et al. (2021) argue that VI. pre-trirggnnng CLIP offers unique

---

Published in Transactions on Machine Learning Research (08/2023)
01.8
0.5
Aoomar Robatnem
smilt
0.7
0.6
mail en Ma beago
0.6
TN INV1CO VT.
VL Velikaiia
IN (CC61
0.5
El Vandaatonn Accenay
VI Avcrage Bokssianc
IN10CCE
Korrage Hebeenon
1N
1/00x
1/80
1/1x
1x
24
10x
20
1000x
10000
Dira Balget (li 120,000 samples)
Scrples Dutrsct
Figure 1: (L) Under data budget, standard CE-loss models oufperform VL-loss models in both accuracy and robustness. (R) For most evaluation metrics, this effect continues in ultra-high data regimes. (L) We train ResNer-50 models using both CF-loss and VL.-loss across wide range of data scales, and find that accuracy of VL-loss and CE-loss models is extremely similar at small scales. For scaling 4X and above, CE-loss models exhibit superior robustness; the CE-loss model trained on just 2.4 Mn JANuS samples has comparable robustness, as well as comparable accuracy, to the CLIP ResNet-50 trained on 400 Mn samples. (See Tab. for information on the JANuS dataset, which create and to train these models, and Sec. 5.1 for a detailed explanation of our methods.) (R) We compare the most robust VL-loss and CE-loss models for every tier of dataset size across three different evaluation metrics. Models trained on fewer than 1.5m samples are trained exclusively on supervised data; larger models are trained on mix of supervised and semi-superve superviee data. VL-loss models are more robust on IN100. CE-loss models are more robust on IN IN1000 and IN100-Doge and when less data is available. See Sec. 5.2 for detailed explanation of our methods. Image best viewed color.
advantages when compared to conventional large-data model training techniques. By contrast, Fang et al. (2022) and Nguyen et al. (2022) argue that VL robustness is a consequence of the training data diversity and quantity, with vision-language pretraining playing little role.
In most real-world applications, data is limited, and unlikely be accompanied by informative natural language captions. For example, the PCam medical imaging dataset from Veeling al. (2019) has only around 320, 000 images. Nevertheless, distributional robustness is of paramount importance in the setting. What can be done to train robust models in data-limited settings without access to informative captions? Can we leverage other attributes of model training which have largely been disregarded in the distributional robustness literature, such as architecture, model size, and image resolution?
1.2 Our Contributions
Our objective in this paper is clearly delineate the potential factors influencing distributional robustness for image classification. To achieve this, we evaluate vast suite of existing models trained on diverse data budgets, supplement- ing with dozens of new models trained from scratch. Overall, our key contributions are as follows:
1. We introduce JANuS (Joint Annotations and Names Set), a new class-balanced dataset with images, labels, and captions. To our knowledge, this is the first such dataset of its kind, (See Tab. 1).
2. We conduct ablation studies of several categories of image classification models using JANuS as a controllable training dataset. For the first time, show that it is possible to train highly robust and accurate models, using conventional cross-entropy (CE) loss. even when both data and model size are limited (See Fig. 1).
3. We conduct the largest meta-analysis (to date) of robustness of image classification models (numbering over 650), including many recent architectures, and show that even with relatively modest model and data scaling (compared to Brown et al. (2020), Radford et al. (2021)), one can achieve robust classification performance. (See Fig. and Fig. 3.)
4. We outline useful heuristics to improve distributional robustness when data budgets are limited. (See Sec. 7).
5. In order to enable future research and reprodueibility, we release our code, our dataset, and a complete enumeration of our results for all models in the study (see supplemental attachments).
2

---

Published in Transactions on Machine Learning Research (08/2023)
2
Related Work
Our paper follows a series of recent works studying robustness under distribution shift in the context of image classi- fication (Recht et al., 2019; Taori al., 2020; Miller al., 2021; Fang et al., 2022; Nguyen et al., 2022). This line of inquiry into distributional robustness focused on the linear fit between in-distribution and out-of-distribttion accuracy found between common image classification datasets (such as ImageNet) and their distribution shifts. In contrast to most of these earlier papers, our analysis takes place in a realistic setting where models are trained on a wide range of datasets. Therefore, following the results in Nguyen et al. (2022), we do not use linear fit measures in our analysis, instead relying on average out-of-distribttion robustness.
Jia et al. (2021); Pham et al. (2021) showed that
distributional robustness possible even as base accuracy approaches otate-of-the-art, as long as sufficient data is available for training. The gains are not limited to CLIP; other VL-loss (vision-language loss) functions also achieve strong distributional robustness (Yu et al., 2022; Wang et al., 2022b). We discuss some of these alternate approaches in Appendix Sec. A, while noting that none of these exhibit superior robustness than CLIP.
The internals of CLIP differ from those of typical models in several important ways: the choice of loss function, the training dataset, and the use of natural language captions as labels. However, identifying which of these differences lead to CLIP's outstanding robustness is still an open question. Recent works have addressed this question from differ- ent angles. Fang et al. (2022) argue that intrinsic diversity of training image data is the main source of the distributional robustness gains of VL models in the zero-shot setting, with factors such as language supervision contributing little to no distributional robustness. However, in different (transfer learning) setting, Santurkar et al. (2022) argue that, given a sufficiently large pretraining dataset and descriptive, low variability captions, contrastively trained VL mod- els are more robust than self-supervised image-only models trained with the SIMCLR-loss. We conduct controlled comparisons between vision-language classifiers and conventional classifiers, and find that when controlling for data quantity and diversity, high accuracy VL-loss models are actually less robust than high accuracy CE-loss models.
Nguyen et al. (2022) is important precursor our work. Their extensive experiments on lision-languag models in the low accuracy regime showed that controlling for the pretraining dataset was essential for understanding dis- tributional robustness. We extend this understanding, and show that model architecture, size, image resolution, and even the label set selected for the classification problem can all have substantial effects on robustness. Finally, unlike Nguyen et al. (2022), all our results are shown in both low and high accuracy regimes, and across different test sets.
In their paper investigating the role of language on robustness, Fang et al. (2022) introduced which added Flickr-captions nearly 450,000 ImageNet images. We extend this work by introducing JANuS, which add over 50,000 new human-supervised samples to 100 classes in ImageNet-Captionn in order to rebalance the classes, as it has been shown that CE-loss models often struggle with imbalanced classes (Phan & Yamamoto, 2020).
3 Preliminaries
Metrics for distributional robustness. Our primary metric is average robustness (abbv: Avg. Rob.), which is the average test-set accuracy of model on a set of distribution shifts. Although this measure is easy to interpret, it can conceal substantial performance differences between shifts. Another metric we use is effective robustness, introduced by Taori et al. (2020), primarily to situate our work within the existing literature. This metric is a graphical tool to describe how robust a model is on natural distribution shifts. For human vision, a graph of versus-shiff test accuracy follows the y = x trendline; for neural networks, this trend-line generally parallel to but below y = x. Finally, we include Effective Robustness Ratio (abbv: E.R.R.), from Feuer et al. (2022) in our appendix tables. This is defined as the ratio of average robustness over base task accuracy. This is an effective measure for comparisons among models with roughly similar base accuracy.
Fine-tuning. Fine-tuning is the extremely common practice of initializing the weights of a model to values attained during pretraining, and then adjusting them based on a new dataset. It has been shown that large-scale pretraining can dramatically improve the base accuracy of computer vision models when compared to random initialization (Doso- vitskiy et al., 2021; Steiner et al., 2022), and that the ImageNet dataset is very well suited to this task (Kornblith et al., 2019). However, distributional robustness generally does not improve in proportion to the gains in base ac- curacy. (Recht et al., 2019) In fact, Radford et al. (2021) found that pretraining and fine-tuning rapidly can erode
3

---

Table 1: The JANuS dataset allows for controlled comparisons between models in high accuracy regime. The experiments in Fig. I (L) were conducted using combination of the four main datasets in JANuS, which are described here. G.T. Lbl. indicates the presence of human-ane auman-annotated ground truth labels in the dataset. Machine Lbl. indicates availability of synthetic labels; labeling strategies are detailed in 4.3. Caption Src. lists the sources for captions in the dataset. Supervised indicates when ground truth labels exist for the dataset. CE-loss models benefit most from Published in Transactions on Machine Learning Research (08/2023)
Dataset
G.T. Label
Machine
La-
Caption
Supervised
Filtered
Balanced
bel
Source
ImageNet-100 (IN100)
Flickr, BLIP
Human
OpenImages- 100 (OI100)
Flickr, BLIP, annotated
None
X
LAION-100 (LAION100)
alt-text
X
CLIP
YFCC- 100 (YFCC100)
Flickr
X
Algo
X
supervised data. Filtered indicates when the dataset contents were processed in some way prior to inclusion. VL-loss models struggle on unfiltered data. Balanced indicates whether the dataset is approximately class-balanced.
distributional robustness, even base accuracy increases. Zhai et al. (2021); Wortsman et al. (2022a;b) closed the gap but were unable to reproduce the zero-shot robustness attained by Radford et al. (2021). Given the limited efficacy of fine-tuning, the majority of our results are reported on models trained from scratch. We confine our specific remarks on transfer learning to Sec. A.
Glossary. For ease of understanding, we provide glossary of common terms and abbreviations.
Loss functions. We examine models trained with two types of losses. VL-loss refers to the InfoNCE loss used by CLIP (Radford et al., 2021). CE-loss is the typical cross-entropy loss used train image classification models.
Label types. CE-loss models use integer labels (referring to discretely labelled classes), and VL-loss models use caption labels. We refer to human-annotatee labels (whenever available) as ground-truth. We refer to labels generated by automated processes as either synthetic or subset-matched (defined below Sec. 4).
Data filrratoon.. We define data filtration as any strategy which sub-selects image-caption pairs.
4 JANuS: A New Benchmark Dataset for Robust Model Training
Training large-scale image classification models from scratch on existing benchmarks may present resource challenges for academic researchers, particularly vision-language models with dual architectures. Prior papers such as Fang et al. (2022) resolve this by obtaining low- or medium-accuraan results, and using the linear fit hypothesis of Miller et al. (2021) to project those results high accuracy regimes. However, as observed in Nguyen al. (2022), trends observed in low-accuracy regimes may persist in high-accuracy regimes, unless the training dataset, loss function, and label set are controlled for across different models.
For this reason, we postulate that 100-class, broad scope problem is ideal for comparative studies of robustness. However, no training dataset exists which is designed for 100-class ImageNet problems and is sufficiently diverse to train high accuracy, high- high-robustness models with both VL-loss and CE-loss objectives.
To resolve this, we introduce JANuS (Joint Annotations and Names Set). a collection of four new training datasets with images, labels, and corresponding captions. Each dataset JANuS builds upon an existing dataset by selecting adding data from known data source. Data sources for which ground truth labels exist are filtered by class. For data sources where ground truth labels do not exist, we use technique called subset matching to prefilter JANuS; detailed explanation of this technique can be found in Sec. H. The constituent datasets are the following:
1. ImageNet-100 (IN100): The 100 largest Imagennt-Captions classes from Fang et al. (2022), followed by class rebalancing by addition over 50,000 new image samples annotated with numan-authored ground-trute -ttnt labels.
2. OpenImages-100 (OI100): A subset of the OpenImages dataset, Kuznetsova al. (2018), with restored original Flickr-captions, and new BLIP-captions; samples selected by mapping commnn-labeled OpenImages- 100 classnames to ImageNet-100 1l classnames.
4

---

Published in Transactions on Machine Learning Research (08/2023)
3. LAION-100 (LAION100); A subset of the unlabeled LAION dataset, Schuhmann al. (2021a), with samples selected via subset matching on ImageNet- 100 classes. YFCC-100 (YFCC100): A subset of the unlabeled YFCC dataset, Thomee et al. (2016), with samples selected via subset matching on ImageNet- classes.
4.
We compare some of the key properties of each component of JANuS Tab. 1.
The most important new contribution in JANuS is ImageNet-100, which is, the best of our knowledge, the only version of ImageNet which duplicates the original distribution's class balance and supervision properties (ImageNet is not perfectly class balanced, but it does not contain any long-tail classes; all classes in ImageNet have at least 750 samples), while also being fully captioned with original web-scraped labels.
Every JANuS sample has descriptive captions as well as class labels (either as human annotated or synthetic labels).
Furthermore, either VL-loss or CE-loss models trained on JANuS alone can achieve high validation accuracy, making possible for the first time compare model distributional robustness while controlling for base accuracy.
These properties enable JANuS to be used fairly compare both image-only and image-text training approaches while controlling for dataset size and quality, making it useful new benchmark for robustness comparisons.
4.1 IN100 performance is comparable to ImageNet.
In order to ensure that the baseline performance of VL-loss and CE-loss models is comparable on IN100 and the stan- dard ImageNet despite the 50,000 newly labeled images, we train a VL (using the standard "A photo of $CLASS- NAME" prompt) and CE-loss model from scratch on IN100, and compare it CE-loss model trained for 256 epochs the same 100-class subset of ImageNet. Controlling for size, we find that dataset performs slightly worse than the baseline, but considerably better than that subset of ImageNet-Captions (Fang et al., 2022) alone.
4.2 Dataset Construction
The 100 classes in JANuS were selected randomly from a subset of all classes with more than 600 captions available in CrageNet-Captions (Fang et al., 2022). The list of classes selected is available in Sec. We acknowledge that as with any filtration strategy, this class selection approach favoring classes with more captioned images may introduce potential selection bias (which might spuriously correlate with accuracy robustness). However, feel that the risk of this is outweighed by the many benefits of having such dataset publicly available for future studies.
4.3 Supervision strategy
In Table in the Appendix, we discuss in detail the supervision (labeling) strategy used for JANuS, with a per-class breakdown of each class. An overview of the supervision process is as follows:
All image samples
supervised by the authors of this paper.
Samples were sourced from Flickr using the available API, sorted by 'interesting'. with safesearch enabled, search- ing only samples with Creative Commons licenses.
Additional filtering terms were passed to the API in order to eliminate common confounds in the search terms.
After the search term was selected, items were downloaded in bulk.
All downloaded samples were then individually tagged by the researchers as either "IN-class" or "out-of-class", using reference photographs from each class as a baseline comparison.
We found that classes varied in several respects:
Some classes had far greater availability than others (ranging from 450,000 to 283 available samples).
Some classes were much cleaner than others (ranging from 100 percent clean to around 25 percent).
Some classes tended to be the 'subject' of images (such as dog breed) while others, such as mashed potato, tended to be featured as secondary items the 'background" of image saple.
5

---

Published in Transactions on Machine Learning Research (08/2023)
5 Experimental Design
Our motivating goal in this paper is to identify which factors are most decisive determining distributional robustness in data-limited regimes. We enumerate several possibilities below:
(Q1) Are models trained more samples more robust than models trained fewer samples?
(Q2) Are models with more parameters more robust than models with few parameters?
(Q3) Are VL-loss models more robust than CE-loss models?
(Q4) Are VIT models more robust than convolution-ban models?
We address each of this questions by designing careful experiments to evaluate model robustness. For each question, we conduct two distinct types of evaluations. Approach 1 is a small-scale evaluation of models trained under highly controlled settings. Approach 2 is a large-scale meta-analysis of publicly available pretrained models. We note that both approaches have strengths and limitations, but believe that trends found to be consistent in both approaches are likely indicative of future model performance. Where trends differ, as in Q4, we try to understand the root cause of these differences.
5.1 Approach 1: Controlled comparisons on JANuS-trained models
In our first set of experiments, we measure accuracy and robustness of various models trained on our proposed JaNUS dataset. Using this approach, we control for confounding factors by changing one aspect of the training reg- imen at time, and we can control for the pretraining dataset. This is important if we are to truly isolate archi- tectural/algorrihmic factors influencing robustness, which has thus far been absent from many existing studies of distributional robustness in the literature'.
In all our model training experiments, we train with mixed precision, at a batch size of 256, and do not use gradient clipping. We use the AMP library to implement the training process. Model hyperparameters are chosen via grid search. Models are typically distributed across single node with 4 NVIDIA A100 GPUs; our largest models were trained on 16 NVIDIA GPUs. All JANuS models were trained for 256 epochs. Following Santurkar et al. (2022), we use SimCLR augmentations (resize, crop, flip. jitter, blur, grayscale) rather than CLIP augmentations (resize and crop) for model training. Our code is publicly available for reproducibility purposes.
Models. We train over twenty-five models on the JANuS dataset; the complete list can be found in Tab. 9. Our baseline model against which all variations are compared is ResNet-50 with a 1000-class linear classification head (He et al., 2015). Specific variations are discussed in more detail below.
Labels. We evaluate single, broad-scope label set of 100 classes corresponding to ImageNet-100 (IN 100),, which is the first constituent dataset of JANuS; see Sec. 4 above. We refer to the validation set for IN100 as INl00-Val.
Shifts. Following Radford et al. (2021), we focus on the following four distribution shifts: Imagenet-Sketch, Imagenet- R, Imagenet-A, and Imagenet- V2, for our evaluation. For ImageNet-R and ImageNet-A, which are subsets of Ima- geNet, we evaluate only the 35 shared classes. We reference the validation sets for these shifts on IN100 as IN100-V2, IN100-S, IN100-R, and IN100-A. Additional details on the datasets, distribution shifts, and class indices are in Ap- pendices Sec. C, Sec. D. and Sec.
Metries We evaluate our models every 32 epochs, and report the "best" average robustness across all shifts, with "best" being determined by the model's 's peak performance across all evaluated checkpoints
With these basics in place, we design experiments specific to questions (Q1)-(Q4).
(Q1) Comparing across dataset size. We report dataset size approximate multiples of the size of JaNUS's IN-100 training set. For example, a 1x JANuS model is trained on approximately 120, 000 samples. A 10x JANuS model is trained approximately 1, 200, 000 samples. We primarily compare models at 1x, 2x, and 10x scales. In addition to directly reporting the effects of scaling dataset size, we report the secondary effects of scaling dataset size when changing parameter count, loss function and architecture.
'Controlling for fixed training dataset has limitations. We cannot evaluate the effects of scaling pretraining data beyond the training size JANuS (2.4 million images); we cannot evaluate effects pretraining on wider range of label classes; and we cannot evaluate models trained on classes outside Net- 100. We address these the larger meta-analysis presented See. 5.2.

---

Published in Transactions on Machine Learning Research (08/2023)
(Q2) Comparing across parameter counts. Our models comparing the effects of scaling parameter count in CE-loss models are ResNet-26, a ResNer-50, and ResNet-50x4 model with identical 1000-class linear heads. The latter is ResNet-50, scaled up 4x according to the EfficientNet scaling rule (Radford et al., 2021; Santurkar et al., 2022).
(Q3) Comparing VL and CE-loss. Our models comparing CE-loss and VL-loss are a ResNet-50, and a CLIP architecture with a ResNet-50 vision backbone. The only difference in the two architectures is that for CE-loss mod- els, we append ResNet-50 with 1000-class linear head, whereas our VL-loss model uses text transformer for classification. Because of this, our vision-language models have significantly higher parameter counts than standard computer vision models; we allow for this difference when controlling for parameter count EG, we compare RN50 using VL-loss to RN50 using CE-loss, rather than comparing the RN50 using VL-loss to a parameter-equivalent RN50x4 using CE-loss. We choose this approach because, as noted in Radford et al. (2021); Santurkar et al. (2022), the difference does not measurably affect distributional robustness.
(Q4) Comparing ViTs and convolutional models. When comparing ViTs to convolutional models, we compare the ViT-S-16 model introduced in Dosovitskiy et al. (2021) our ResNet-50 baseline. ViT-S-16 has approximately 88% as many parameters the ResNet-50. We do not ablate the effects of this difference architecture performance, but our scaling experiments across parameter counts lead us to believe that the effect is minor, compared to other factors.
5.2 Approach 2: Large-scale Meta-Analysis of Pre-trained Models
Approach 2 is a large-scale meta-analysis of publicly available pre-trained models, similar to the setting previously adopted in Taori et al. (2020); Miller et al. (2021). This approach illuminates the effects of scaling pretraining data across large ranges, as well as the effects of pre-training on wide range of classes. While this enables to compare trends across hundreds of models, we note that such observational studies cannot carefully control for architectural and/or algorithmic confounding factors. Our complete results can be found in our main results table in the supplemen- tary attachment, 1_ captionnet inlk model_results and metadata. Our experiments are designed as follows.
Models. We compare over 650 models drawn from the pytorch-image-modee (timm) repository (Wightman, 2019), the open-clip repository (Ilharco et al., 2021), and a few newly trained models². Architectural details about the models included in this approach are listed in Sec. G. We conduct additional fine-tuming of these models.
Labels. We report evaluation results on the label set of ImageNet-IKK, which is the standard benchmark in the majority of existing works. We refer to this validation set as VNl. In addition, we also report results on the label set of ImageNet-100 100 (defined above), which is a broad scope label set; we also evaluate on a fine- aneiggrined subset of ImageNet- IK that we call ImageNet-Dogs BNN00-Dogs, for short). This consists of subset of ImageNet class labels corresponding 100 dog breeds, and we do this in order to provide an alternative view of 100-class classification using ImageNet models. We refer this validation set as IN100-Dogs-Val.
Shifts. We utilize the same four shifts as in Approach 1. We abbreviate these shifts following the same pattern described in Approach 1; IN1000- V2, IN100-Dogs- V2, IN1000-S, IN100-Dogs-S, etc.
Metrics The metrics are identical to those used in Approach 1, except that we provide the average for each label set individually, rather than reporting the aggregate over all label sets. We do not report the average of all models fitting a certain category, as that would skew the evaluation against older releases in the repository. Instead, we report the average of the ten highest performing models in each category. For example, if we are evaluating the effects of parameter count, we first bin the models by parameter count, and then report the average of the ten best performing models in each bin. We also present individual model scores in scatterplot format in our figures.
6 Results, Trends, and Ablations
We now present a series of experimental results that address each of the questions Q1-Q4 listed in Sec. 5. For each question, we present our key findings, and then support these from our experimental results with Approach I (controlled experiments on JANuS) and Approach 2
with pre-trained models.)
We intended restrict our meta-analysis models available existing public repositories; however, there seems a dearth of models trained with number of samples between 15M and 300M. We fill this gap by training several models increasing size subsets of the LAION-2B dataset.
7

---

Published in Transactions on Machine Learning Research (08/2023)
Avg. robustness by label set
Model
IN1000 Val.
IN1000 Avg. Rob. IN100 Avg. Rob.
INg00-Dogs Avg. Rob.
VOLO-D5-224 (1.2 Mn)
.857
.594
.725
.723
VGG-16 (1.2 Mn)
.716
.266
.402
.433
Avg. robustness by shiftt
Model
V2
VOLO-D5-222 (1.2 Mn)
.814
.55
.652
.707
VGG-16 (1.2 Mn)
.66
.251
.363
.195
Table 2: Choice of architecture can strongly impact model performance controlled-data settings. contrast recent claims in the robustness literature, we find that factors other than data can strongly affect distributional robustness. Specifically, when we compare VGG-16 (Simonyan & Zisserman, 2014) baseline recent VOLO model (Yuan et al., 2021), we see substantial gains in robustness. The gain is not localized to particular shifts, nor does it scale in proportion with base accuracy.
Averar Roborrsem
0.7
0.5
0.6
0.5
RNDS Ruhaatess
Vaidatie Acuusey
0.4
A R300 surrag Ramarranss
R2S004
Valinice Azcanzy
RN90el Aurage Remaetais
1x
2x
Rx
Bedget wngplei
Label sct
Figure 2: (L) Effeets of scaling parameter count (Approach I). (R) Effects of scaling parameter count (Approach 2). (L) We find that increasing the parameter count has a positive effect on average robustness, but the effect is limited on small data budgets, and can invert when data is very limited. (R) When we compare the average robustness of models in the study by their parameter count using approach 2, we see reliable improvements as model size increases; larger models are more robust on all label sets. Image best viewed in color.
(Q1) Are models trained on more samples more robust than models trained on fewer samples?
Increasing quantity of training data is known positively impact robustness. Fang et al. (2022) argued that diverse, and presumably large, training distributions nearly exclusively account for the strong robustness of VL-loss models, and Taori et al. (2020); Miller et al. (2021) contend that among CE-loss models, factors other than data have little impact on robustness, except insofar as they increase base accuracy. If so, then we should expect that data (and data alone) can impact distributional robustness.
The findings in Fang et al. (2022) relied on prior work from Taori et (2020) in order to project their findings from low high accuracy models. However, the work of Taori et al. (2020) predates the introduction of some unusually robust architectures, including Yuan et al. (2021); Hatamizadeh et al. (2023); Liu et al. (2022). The linear trend in the accuracy/robustness space described in that work no longer holds for all shifts and all models, even when controlling for pretraining dataset, we document this in (Q4), as well as in Appendix Tab. 10, Tab. 13, Tab. 11.
Overall, we find that although it is not the only factor, dataset size is still an important determinant of distributional robustness.
Evidence using Approach 1. We find that increasing the quantity of training data has reliable and positive effect on average robustness. Specifically, baseline model at 1x scaling averages .42 robustness, whereas with 2x scaling, it
8

---

Published in Transactions on Machine Learning Research (08/2023)
improves to .51. scaling diminish at the extremes. See Table 9 for additional results in this vein.
The marginal improvements decline 10x (.66) and 20x (.7) scales, suggesting the returns of data
Evidence using Approach 2. This finding persists when we make large-scale comparisons of over 650 pre-trained models. For example, a standard ResNet-50 model trained on ImageNet- IK (i.e., a 10x data budget in our terminology) achieves a robustness of 0.504. The SWSL. ResNet-50 model trained by Mahajan
(2018) on 7800x data budget achieves robustness of .74. The full table is too large to parse; please refer to the supplementary attachment, 1 capptionnett in model results and metadata.
(Q2) Are models with more parameters more robust than models with fewer parameters?
Earlier works such as Mahajan et al. (2018); Radford et al. (2021); Tan & Le (2019) have shown empirically that parameter count, in conjunction with other factors, can affect both validation accuracy and distributional robustmess. However, Miller et al. (2021); Taori et al. (2020) found that when pretraining on ImageNet, increasing parameter counts improved robustness only within certain predictable limits.
Evidence using Approach 1, Our findings here agree with both groups; on JANuS, we find that increasing the parameter count has positive effect on average robustness, but the effect limited on small data budgets; when dataset size very limited (Ix or below), increasing parameter count can actually decrease robustness. (see Fig. 2)
Evidence using Approach 2. In analysis. 438 of the models have fewer than 50 million parameters, 126 have between 50 and 100 million, and 86 have over 100 million parameters. Of the top 100 most robust models, 13 have fewer than 50 million, 33 have between 50 and 100 million, and 54 have over 100 million parameters.
In Fig. 2 (L), we compare model performance on our three evaluation metrics, grouped by parameter count. We find that average robustness improves reliably with model size across all evaluations. The gains are most significant on IN100.
(Q3) Are VL-loss models more robust than CE-loss models?
Large VL-loss models such as those of Radford et al. (2021); Pham et al. (2021) have been conventionally presented as robust generalist models which can handle arbitrary (open vocabulary) classification tasks. However, to our knowledge large-scale validation studies quantifying the value of VL-training in terms of robustness have been reported thus far in the literature. We fill this gap.
Overall, when controlled for other factors, we find that VL-loss models are no more robust than CE-loss models.
Evidence using Approach 1. We train VI.-loss and CE-loss models from scratch on JANuS and evaluate them on IN100 across wide range of data scales.
Ground-true labels have been shown to improve base accuracy of VL-loss models. Fang et al. (2022) found that ResNet-50 VI--loss model trained on ImageNet- k with ground truth labels ("A photo of the CLASSNAME") achieved accuracy and robustness parity with a CE-loss ResNet-50 for IN1000 classification. In Fig. 1, we show that this is also the case for ResNet-50 models trained and evaluated on IN100.
However, we see that at no point does VL-loss offer a robustness advantage. On the contrary, at larger data scales, VL-loss is at disadvantage; a CE-loss model trained on just 2.4M JANuS samples has comparable robustness, as well as comparable accuracy, to CLIP ResNet-50 trained on 400M samples.
Evidence using Approach 2. For dataset sizes below 400M samples, we find reliable evidence that VL-loss models more robust than CE-loss models in absolute terms IN1000 IN100-Dogs, Fig. 2 (R). We also note that VI.-loss models have lower base accuracy on these problems. VL-llss models do show robustness advantage on IN
*e postulate that this discrepancy may be because smaller label sets casier to disambiguate using natural language, and provide two experiments evidence. First, in Sec. F, provide per-class accuracies for VL-loss and CE-loss ResNet-50 trained many samples, and note that several of the classes IN1000 where VL-loss models substantially underperform CE-loss models have identical natural language descriptions; in IN1000, OpenAl's classnames include two classes labeled "missile" and classes labeled "sunglasses" reflecting ambiguities the underlying problem (Radford et al., 2021; Beyer al., 2020). Second, train VL-loss model million YFCC samples, filtering out samples whose caption contains matching term with ImageNet class; the resulting model has, in theory, not been trained any of the classes on which

---

Published in Transactions on Machine Learning Research (08/2023)
Above 400M samples, our investigation is limited by the fact that relatively few public models have been trained such huge datasets; our largest CE-loss models were trained on half the data of the largest Vl.-loss models, and they have few other architectural features in common. The limited evidence we have indicates that VL-loss models might enjoy robustness advantage at very high data regimes (See Fig. 1); however, this might be confounded with the fact that no publicly released ViTs have been trained with CE-loss at this scale.
Model
Dataset
IN1000 Val. Acc.
IN100 Val. Acc.
IN100-Dogs Val. Acc.
ResNet-50
YFFCC-10M-NNI. YFCC-15Mn
.127
.329
.034
ResNet-50
.324
.741
.086
Table 3: VL-llos models trained on web-scraped caption labels learn classes unevenly. VL-loss models learn a lot about broad distinctions between classes from captions, and little about fine-grained class boundaries. This finding holds even when we remove all samples which match with any term in the OpenAl ImageNet classnames from the YFCC-15Mn dataset (YFCC-10Mn-N.I) Robustness scores can be found in our main results table in the supplementary attachment, __cpptionnnt_ttt inlk model results and metadata.
(Q4) Are ViT models more robust than convolution-based architectures?
In Tab. 2, we demonstrate that the marginal robustness gain of going from VGG-16 from Simonyan & Zisserman (2014) to VOLO-D5-224, the best model using Approach 2, trained ImageNet-IK 1k alone, at 224px image resolution, is much greater than the gains of data scaling alone. This finding motivates a consideration of which architectures are likely to be more robust, given data budget. One natural way to divide approaches is to compare models based on the vision transformer architecture to convolution baaset approaches.
Overall, we find that architecture can strongly impact distributional robustness, controlling for all other factors.
Evidence using Approach 1. We observe that ViT-S-16 performs worse than the parameter-equivalent RN50 JANuS. This finding is consistent from the smallest to the largest scales consider. (see Fig. 3) This finding is in keeping with Dosovitskiy et al. (2021), who found that ViTs only reach parity with CNNs after very large scale pretraining. Unlike Dosovitskiy et al. (2021), we find that pretraining and fine-tuning does not necessarily help close the gap. ViT' are more robust and more accurate than ResNets after pre-training but before fine-tuning. After fine tnning, the advantage disappears. (see Tab. 4)
Dataset
Data Scale
Fine Tune
RNSO IN100
RN50 IN100
VIT-S-16 IN100 VIT-S-16 IN100 Avg. Rob.
Val. Acc.
Avg. Rob.
Val. Acc.
IN100 (JANuS)
1x
FALSE
0.87
0.424
0.713
0.256
IN100+01100 (JANuS)
2x
FALSE
0.904
0.511
0.785
0.325
IN100+LAION100 (JANuS)
4x
FALSE
0.901
0.57
0.787
0,376
INIk
10x
FALSE
0.956
0.505
0,942
0.474
INIK+NN100
10x
TRUE
0.937
0.564
0.926
0.502
JANuS
10x
FALSE
0.908
0.655
0,823
0,464
JANUS+YFCC
20x
FALSE
0.927
0.702
0.878
0.576
IN21KINIK
100x
FALSE
0.939
0.538
0,957
0.543
1N2IK-NNKH
100x
TRUE
0.924
0.499
0.926
0.501
Table 4: VITs are usually less accurate and robust than ResNets on data budgets. ViTs consistently underperform ResNets across a wide range of data budgets. After pretraining but before fine-tuning. ViTs are more robust and more accurate than ResNets. After fine tuning, the advantage disappears. This indicates that the robustness advantage enjoyed by massively pretrained ViT's may not be preserved during fine-tuning.
Evidence using Approach 2.
The 650 models in our analysis include 385 convolution-based architectures and 204 vision transformers; despite the relative overrepresentation of convolution architectures in the study, the 100 timm
evaluate it. When do this, find that the resulting model achieves just accuracy IN100-Dogs, but achieves 33% accuracy on IN100; in absence ground muth labels, model "guesses better', essence, when classes dissimilar. See Tah.
10

---

Published in Transactions on Machine Learning Research (08/2023)
0.3
Avenge Averge Reivoaees
67
64
6.-
Cvenckikas ACSEIO
Connttatinn fesed aditsurr
Saliburee Aczerac AT Arang: Rizaaco
06
Daga
Tao Bldge 1N100 coeples
Like sct
Figure 3: (L) Effects of architecture (Approach 1). (R) Effects of architecture (Appreach 2). (L) We compare average robustness of ViT-S-16 and ResNet-50 on JANuS, using a range of data scales, and find that the ViT underperforms the ResNet throughout.
(R) We evaluate average robustness of models the study when grouped by architecture, and find ViTs outperform convolution based architectures. Individual marks on the graph represent the average robustness of the ten most robust convolution- based and transformer-base models, respectively. Trend lines follow the group average. Image best viewed in color.
models with the highest average robustness on IN1000, 60 are ViTs and 40 are convolution-base architectures. On IN100, the split is 70/ 30, and on IN100-Dogs, 72/ / 28.
Comparing the top 100 most robust models for each problem, we find that ViTs are, on average, substantially more robust, and the advantage grows massive data scales (see Fig. 3).
Interpreting the trends. Aside from the aforementioned limitations of VITs when training limited data, attribute the strong performance of VIT's in Approach 2 to the recent emergence of new variations on the ViT architecture which are more robust than the standard ViTs. Recent effective interventions have modified the attention mechanism Yuan et al. (2021); Hatamizadeh et al. (2023) or the pretraining strategy Bao et al. (2022); Touvron et al. (2022) to achieve substantial gains in this regard. See Tab. 5 for further details.
Model
IN100-Val
IN100-V2
IN100-S
IN100-R
IN100-A
IN100 Avg. Rob.
11100-ViTS16
0.713
0.584
0.124
0.204
0.11
0.256
in100-DerTS16
0.756
0.643
0.164
0.222
0.137
0.292
in100-GCViTSI6
0.803
0.702
0.378
0.356
0.143
0.395
Table 5: Recent ViT-hased architectures are more robust and accurate on data budgets. Although they do not match the performance of ResNets, recent improvements to the ViT such as DeIT from Touvron et al. (2022) and the CCT from Hatamizadeh et al. (2023) improve dramatically on standard ViTs. All models listed here were trained on IN100, at a 1x data budget, for 256 epochs.
7 Discussion and Useful Heuristics
7.1 Summary of Key Findings
Our detailed experimental results in Section demonstrate the effects of various architectural, algorithmic, and data- dependent factors distributional robustness of image classification models. We conclude with a summary of key takeaway points, along with list of suggested useful heuristics for training robust models in future applications.
1. Motivated by our findings for Q1, scaling data quantity is likely to have the most reliable impact on distributional robustness during training.
2. Motivated by our findings for Q3 (Approach 2), for few-class problems where either the classes themselves or their visual properties (color, shape, etc.) are easily disambiguated using text alone, we conclude that the most robust and most efficient approach is zero-shot VI. model.

---

Published in Transactions on Machine Learning Research (08/2023)
3. Motivated by our findings Q2, Q3 and Q4, for fine-grained classification problems, for problems with ambiguous class names and many-class problems, the best approach when training from scratch is a CE-loss model with tailored convolution-based architecture such as ConvNeXt, scaled to parameter counts at which returns diminish.
7.2 Future Work
As computer vision models and datasets grow in size, and multimodal generative models such OFA from Wang et al. (2022a) introduce and solve new, complex problems, the task of developing prescriptive set of "scaling laws" for emergent distributional robustness will only increase in importance (Cherti et al., 2022). Equally important will be comparing the behavior of models on distribution shifts for datasets other than ImageNet. Finally, a comprehensive understanding of model performance on more challenging. long-tailed classification problems (such as iNaturalist) will shed more light on the robustness profile of models in the real world.
Acknowledgements
This work was supported in part by the National Science Foundation (NSF) under grant CCF-2005804, an Al Research Institutes grant supported by NSF and USDA-NIFA under the AI Institute for Resilient Agriculture grant 2021 1021017021- 35329, and the Department of Energy under grant 1E-EE0009105.
12

---

Published in Transactions on Machine Learning Research (08/2023)
References
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ICLR, 2022. 11,
Lucas Beyer, Olivier J. H' enafff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? NeurlPS Workshop: ImageNel Past, Present, Future, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurlPS, 2020. 2
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. CVPR, 2021, 20, 32
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. NeurlPS, 2017. 32
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive imnuuaee-iagee learning.
ArXiv, abs/2212.07143, 2022. 12
Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Arxiv, 2021. 32
Stéphane d' Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. Convit: improving vision transformers with soft convolutional inductive biases. Journal of Statistical Mechanics: Theory and Experiment, 2021. 32
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, 2009. 19, 37
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition scale. ICI.R, 2021. 3, 7, 10
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), ICML, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceedings of Machine Learning Research, 2022. 2, 3, 5, 8, 9, 19, 32, 37
Benjamin Feuer, Ameya Joshi, and Chinmay Hegde. meta-analysis of distributionalll-rrouss models. ICMI PODS Workshop on Distribution Shift, 2022.
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's ' clothing for faster inference. ICCV, 2021. 32
Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing Xu, and Tong Zhang. Model rubik's cube: Twisting resolution, depth and width for tinynets. NeurIPS, 2020.
Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global Context Vision Transformers, June 2023, URL http: www.wwwwwwwwwa.o/oo/opaaaaaaaaaa 2206. 09959. MrXiv:2206.09959 [cs]. 8, 11
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1511.03388,, 2015. htthtt/:/ http::///xxvvx..//aab.. /1522 03385. 6, 32
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and pertur- bations. ICL.R, 2019. 1
13

---

Published in Transactions on Machine Learning Research (08/2023)
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a. 19
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b. 19
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. CVPR, 2016. 32
Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL
bttps::/dof orgg 5281111/./.11 20t8////enoo.5111 5143773.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- lision-language representation learning with noisy text supervision. In ICML, 2021. 1,3
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby, Big transfer (bit): General visual representation learning. In ECCV, 2019. 17, 32
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do Better ImageNet Models Transfer Better? In CVPR. arXiv, June 2019. doi: 10.48550/arXiv,18000 1805.08974 URL httt:://arxiv.orr//bbee aa0r0/ 08974. arXiv:1805.08974 [cs, stat]. 3
Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision. 2018. 4
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision, 2020. 37
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022a. 31, 37
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Effi- cientformer: Vision transformers at mobilenet speed. NeurlPS, 2022b. 32
Zhuang I. Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. CVPR, 2022.
Muhammad Maaz, AbdeIrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muhammad Anwer, and Fahad Shahbaz Khan. Edgenext: Efficiently amalgamated enn-transformer architecture for mobile vision applications. In International Workshop on Computational Aspects of Deep Learning at 17th European Conference on Computer Vision (CADL2022), 2022. 32
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe. and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Confer- ence, Munich, Germany, September 8-14, 2018, Proceedings, Part II, volume 11206 of Lecture Notes in Computer Science, pp. 10077978- 3- 030-0112000 -1216-811111161
Springer, 2018. doi:
12. URL https: /tw..//oiii./////// http:///oo..ogg...//// org/10.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. ICMI., 2021. 1, 4, 7, 19
14

---

Published in Transactions on Machine Learning Research (08/2023)
Abdul Arfat Mohammed and Venkatesh Umaashankar. Effectiveness of hierarchical softmax in large scale classi- fication tasks. In 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI). IEEE, sep 2018. doi: 10. 10011009ccccc22208855555677 21
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. NeurIPS, 2022. 3, 4, 19
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer learning. ArXiv, abs/2111.10050, 2021. 3,3,
Trong Huy Phan and Kazuma Yamamoto. Resolving class imbalance in object detection with weighted cross entropy losses. arXiv, 2020. 3
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In ECCV, 2020. 31
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021. 1, 2, 3, 4, 6, 7, 9, 17,
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML,, 2019. 3,
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a caption worth a thousand images? controlled study for representation learning. ICLR, 2022. 3, 6, 7, 20
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. Data Centric NeurIPS Workshop, 2021a. 5, 20,
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. ArXiv, abs/211 1.021 14, 2021b. 32
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.11566, 2014. 8, 10
Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. CVPR, 2021.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your VIT? Data, Augmentation, and Regularization in Vision Transformers. TMLR, June 2022.
URL
bttt:://arxvve o
10270. arXiv:2106.10270 [cs].
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019. 25, 32
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. NeurIPS, 2020. 3, 7, 8, 19
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl S. Ni. Douglas N. Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Commun. ACM, 59:64-73, 2016. 5, 20, 32, 37
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. ICCV, 2021. 32
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. ECCV, 2022. 11, 32
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. CVPR, 2021. 32
15

---

Published in Transactions on Machine Learning Research (08/2023)
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant CNNs for digital pathology. In MICCAI, 2019. 2
Haohan Wang, Songwei Ge, Eric Xing, and Zachary Chase Lipton. Learning robust global representations by penalizing local predictive power. In NeurlPS, 2019a. 19
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recogni- tion. CVPR, 2019b. 32
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-ssqquene learning framework. CoRR, abs/2202.03052 2022a. 12
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. ArXiv, 2022b. 3
Ross Wightman. Pytorch image models. GitHub repository, 2019. doi: 10.5281/2emodo.4414461. 7.32
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, 2022a. 4, 17
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero- shot models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7959-7971. June 2022b.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architec- ture search. CVPR, 2018. 32
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V. Le.. Self-training with noisy student improves imagenet classification. CVPR, 2020. 32
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. ICCV, 2021. 32
Ismet Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Kumar Mahajan. Billion-scale semi- supervised learning for image classification. Arxiv, 2019.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. CVPR, 2017. 32
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 2021. 8, 11
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-imad image text tuning. ArXiv, abs/2
2021. 4, 32
Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Sercan Ö. Arik, and Tomas Pfister. Nested hierarchical trans- former: Towards accurate, data-efficient and interpretable visual understanding. In AAAI Conference on Artificial Intelligence, 2021. 32
16

---

Published in Transactions on Machine Learning Research (08/2023)
Imagenet v Average Robustness: VL. LiT-Tuning. Wise-FT
04
LiT-Tuning
LTI-Tuning from VL
03
Wise-FT
VL: R2=0.98
06
LIT-WIseFT: R2=0.80
Average Robustness
Human robustness
02
+
ImageNet Base Accuracy
Figure 4: Wise-FT, optimized to balance id/ood accuracy, fits the LiT-tuned effective robustness line, Both Wise- FT' and LiT-tuning exhibit lower effective robustness than conventional vision-language pretraining.
A Transfer learning in vision-language models
Another approach to robust classification in VI. is using some form of transfer learning instead of training from scratch. The robustness advantages transfer learning are well understood in conventional computer vision (see Kolesnikov et (2019)), and many recent model releases include variants which are pretrained on ImageNet-21k: Such models generally exhibit improved robustness when compared to models trained on ImageNet-Ik 1 alone (See main table in supplemental attachments).
There are few prominent strategies for transfer learning in VL-loss models as well; catalog them below and discuss their strengths and weaknesses.
Fine-tuning VL models. when the image tower alone fine-tuned. As reported in Radford al. (2021), fine-tuning the VL-loss vision tower using CE-loss objective improves base accuracy degrades robustness. This effect grows stronger the longer the model is fine tuned, making fine-tuning the image-tower an inefficient solution for problems where robustness is consideration.
Unfortunately, the unique robustness properties of VL-loss models are not conserved
A similar effect takes place if both vision and language towers are fine-tuned on ground-truth caption data; after epochs of fine tuning on IN1000, a ViT-L-14 CLIP base accuracy improves from .76 to .8;; however, average robustness declines from .72 to .69. (See main table in supplemental attachments).
Wise-FT, introduced by Wortsman al. (2022a) is fine-tuning method which interpolates the weights of zero-shot CLIP with its fine-tuned counterparts. For certain distribution shifts, is possible find 'sweet spot' where both i.d. and o.o.d. accuracy increase. However, Wise-FT models lose zero-shot capability, and are still not as robust VL-loss models with the same base accuracy.4
LiT-tuning, LiT-tuning, or locked-image text-tuning, is an alternate approach to vision-language training in which a pretrained image tower aligned with an untrained language model. LiT-tuned models are somewhat more data- efficient than VL models trained from scratch, but they, too, are not as robust as VL-loss models with the same base accuracy. (See 4).
Additionally, observe the following:
17

---

Published in Transactions on Machine Learning Research (08/2023)
Imagenet Average Robustness: VIT-8-32
wiseft
lit
conventional
04-
Average
ImageNet Base Accuracy
Figure 5: LiT-tuning on VL-trained image tower reduces accuracy without altering effective robustness, sug- gesting that VL pretraining is at least robust LiT-tuning. Wise-FT tuning greatly increases base accuracy and slightly improves effective robustness, the cost of zero-shot capability. CE from-scratch training matches Wisc-FT accuracy, but sacrifices effective robustness and zero-shot.
1.
Like Wise-FT, LiT--uuiin produces models whose i.d. / 0.o.d accuracy trade-off fits line between that of traditional models and VL models more robust than the former, less robust than the latter. The only exception we found was when we LiT-tuned the vision tower of a ViT trained on the CLIP objective in this case, LiT-tuning decreased base accuracy while holding effective robustness constant (the npar-opposite effect of Wise-FT)
LiT-tuning offers negative benefit for fully trained VL models, suggesting that it can only hope approach. rather than exceed, the accuracy of its baselines (See 5)
3. LiT-tuning performance tends to closely correlate the base accuracy of the underlying vision model
4. Intriguingly, we find that this is true regardless of the specific dataset used for LiT-tuning LiT-tuned models trained on small amounts of data are able to recover accuracy out-of distribuuto tasks even when very little data from that distribution shift appears in the pretraining data
5. These experiments suggest that some degree of effective robustness is "locked away" in many vision models, but lost during the training process, but that certain techniques are able to increase effective robustness disproportionate to the loss base accuracy, pushing the model 'above the line" we would normally expect. Furthermore, if the distribution shift of interest is known and well-defined, is possible to select a tuning optimize for that shift
B Additional Findings Regarding Transfer Learning
However, as the commonly used robustness evaluations are themselves derived from ImageNet, it is difficult to know to what extent model performance dependent on class-specific ImageNer pretraining rather than general pretraining.
To better understand this distinction, we conduct an ablation study on the pretraining dataset; the results can be found in Tab. 6.
18

---

Published in Transactions on Machine Learning Research (08/2023)
Pretraining Dataset
Num Pt Classes
Data type
Includes IN100 Classes
IN100 Val.
IN100 V2
IN100 S
IN100 R
IN100 A
Avg. Roh.
EIF. Rob. Rat.
Scratch
NIA
NIA
NZA
0.693
0.584
0.184
0.224
0.122
0.279
JANoS
100
Natural
T
0.698
D.588
0.208
0.237
0.128
0.29
0.416
Fractals
1000
Syathetic
F
0.729
0.623
0.239
0.268
0.126
0.314
0.43
INIK-mimIs-NN00
1000
Natural
F.
0.741
0.649
0.255
0.277
0.144
0.331
0.447
INIK
1000
Natural
T
0.771
0.683
0.295
0.314
0.144
0.358
0.465
Table 6: Gradations in performance as a result of pretraining dataset choice. Even when holding the size of the pretraining dataset constant, as we do in this experiment, we find that many factors affect the downstream performance of models. The presence of training classes in pretraining data is impactful; tilk-minus-innO0 pretraining, where we eliminate the in classes, causes a 3% drop in aceuracy, compared to the in Inlk model. Pretraining on a purely synthetic dataset of I Mn. fractal images is surprisingly effective, resulting in only a 4% drop in accuracy. JANuS pretraining is the least effective, comparable to training from scratch. This may be attributable to the fact that JANuS has fewer classes, or to the fact that JANuS has more label noise.
C Distribution Shifts
ImageNet is large-scale visual ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images, making it roughly class-balanced, fully supervised dataset. (Deng et al., 2009)
ImageNet-2Ik, the largest version of ImageNet, contains 14,197,087 images in 21,841 classes.
There now exist a wide range of distribution shifts on ImageNet. These are novel test datasets designed to overcome some of the limitations of the original benchmark. While they cannot remedy issues with the labeling scheme, these datasets do provide challenging new contexts in which to analyze classifier performance.
ImageNet-V2 was designed duplicate, as closely as possible, the original ImageNet test set. It was intended to answer the question of whether ImageNet-trained classifiers could successfully generalize even to the most mild of distribution shifts. (Recht et al., 2019)
Imagenet-Skette is distribution shift covering sketches, paintings, drawings and illustrations of ImageNet classes. This test set is very large and comprehensive. (Wang al., 2019a)
Imagenet-R is a 200-class subset of ImageNet-2012 focused on renditions of everyday objects, defined broadly as drawings, paintings, photographs of food art, etc. (Hendrycks et al., 2021a)
Imagenet-A is 200-class subset of ImageNet-2012 which was algorithmically selected the natural distribution shift captured here is the set of ImageNet-class images which most often fool a RN50. This test is challenging, and tends to include a lot of images with challenges such as occlusion, changes in angle or position, and changes in brightness. (Hendrycks et al., 2021b)
C.1 Different shifts respond to different interventions
Recent works such as Fang al. (2022) demonstrate the power of effective robustness as an explanatory tool for performance differences in VI. models; Miller et al. (2021) showed that there exists strong correlation between most models trained on random subsets of a data distribution, and the fully trained model. However, these authors also caution that it has significant limitations - Taori et al. (2020) and Nguyen et al. (2022) show that models trained on more (or different data) can significantly change the effective robustness line of a particular model, and also that these changes were spectt-peecfic, with stronger fits on shifts like ImageNet- V2 and weaker fits on shifts like ImageNet-A.
We find that ImageNet-YY V2 responds more to model architecture than other shifts, with the handful of -es-Reteet models we evaluated outperforming nearly all other models, regardless of training objective.
ImageNet-R and ImageNet-Sketch both showed high sensitivity to the training data, with the CC12M and LAION- distributions considerably outperforming even the best YFCC-trained models. These types of shifts are particularly amenable to subset matching Seategies.See Fig. 13, Fig. 10 for examples.
On ImageNet-A, CE models significantly underperformed compared to VL models regardless of the data, and all models significantly underperformed compared to the VIT-I CLIP. See Fig. 11 for examples.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Model Name
Average Validation Accuracy
Average Robust Accuracy
CLIP-RN50 (1000-class)
0.5985
0.4306
CLIP-RN50 (Avg. 100-class)
0.8517
0.7182
SWSL-RN50 (1000-class)
0.8362
0.6857
SWSL-RN50 (Avg. 100-class)
0.9524
0.7612
Table 7: Zero-shot model robustness is affected by the difficulty of the task. Both quantity and quality of labels alters model accuracy and robustness under shift; also changes the comparative performance of VL-loss and CE-loss models. In this table, transform the 1000-class IN1000 label set into ten 100-class label sets, and find that the resulting predictions are far more accurate and robust, particularly those of the VL-loss model. This finding motivates our choice to study model robustness on multiple label set sizes.
We also note that there is no readily apparent logit-scaled linear trend in these distribution shifts when one consid- ers models trained on a wide range of different datasets, underscoring the importance well-chosen baseline for comparison.
We find that different shifts tend to disadvantage diffferent kinds of models, which makes improving on all of them simultaneously very challenging. The fact that ViT-L CLIP able to do is both impressive and, given the vital importance of the underlying data distribution in such measures, mystery which is unlikely to ever be solved. Even the massive public datasets such as LAION are unable to match the performance of the dataset CLIP was trained on, although other factors might possibly have played role.
A standardized benchmark of distribution shifts on ImageNet would be a welcome contribution to this area research.
D Pretraining Datasets
Today, many SOTA models are pretrained on web-scale unsupervised data. We utilized three such datasets in our experiments. We observe that one major challenge conducting research on unsupervised datasets is that the links provided as part of the dataset fail more and more over time, leading each group getting different version of the dataset. Therefore, the extent possible, we report the details of each dataset in the appendix, and encourage other researchers working with these datasets to do the same.
CC-12M is lightly supervised web-scale dataset created by Google. The image-caption pairs in CC-12M were filtered and selected for the purposes of training models to caption images. (Changpinyo et al., 2021) Our version of CC12M contained 9703885 image-caption pairs.
YFCC-15M is a subset of YFCC-100M, which 100M image- metadata pairs taken from Yahoo-Flickr 2016. The subset was selected by OpenAL. This dataset contains images and metadata, which includes a "title" and "description" field. These fields combined and processed in various ways by researchers in order to generate captions for models to train on. (Thomee et al., 2016) Our version of YFCC contained 14825134 image-caption pairs.
LAION is a 5B image-caption dataset recently created by ai. It is the first publicly available dataset which matches the scale of the datasets used by the large companies to train their best models. (Schuhmann et al., 2021a) The subset of LAION we refer as LAION-15m contained 13775512 image-caption pairs.
E
Data Quality: Other Considerations
There are various other important considerations, aside from the raw count of per-class samples, in determining the utility of dataset. We follow Santurkar al. (2022) in referring to these as data quality considerations.
We proceed to discuss some of these considerations.
20

---

Published in Transactions on Machine Learning Research (08/2023)
E.1 Image Size
In Fig. 12 (R), we plot average robustness against image resolution, expressed as the ratio of actual model resolution to maximum model resolution in the study (800px). We find that increasing input image resolution leads to gains in robustness on IN1000 and IN100-Dogs, but that these effects are smaller than choice of architecture and number of model parameters.
E.2 Class Imbalance
Aside from simply considering the overall size of dataset, it is important to also consider the per-class size of dataset.
The significance of this distinction can easily be seen on JANuS, when we compare the performance of O1100-trained and IN100-trained models Tab. 9.
Despite the fact that 01100 is slightly larger than IN100, models trained on O1100 perform worse on IN100-Val than models trained on IN100. We find that the extreme class imbalance shown in Fig. E.3 is the cause of most, but not all, of the decrease in accuracy.
VL-loss class imbalances (detected by searching for exact-match classnames in caption strings) are also present in the other web-scraped datasets in JANuS, LAION and YFCC; this may contribute to the lower performance VL-loss models on long-tailed classification.
E.3 Label Set Size
One important, but rarely considered, factor in distributional robustness is the size of the label set.
Specifically, we find that models trained on many-class problems become more accurate and robust when the label set size is reduced to subset of those classes at inference time, and the improvements are not necessarily proportionate
Very large label sets have previously been shown cause declines in base accuracy. (Mohammed Umaashankar, 2018) In Tab. 7, show that this effect present when only 1000 classes used (the size of ImageNet), and that it affects distributional robustness well base accuracy.
Based on this observation, we recommend taking into account label set size when evaluating model robustness, and propose reducing the size of the label set as an effective intervention for improving robustness.
21

---

Published in Transactions on Machine Learning Research (08/2023)
oi100 i01 class distributions
count pi100
count in100
50003
10000
5003
1003
100
todl parpper
bope
Inwo Sloppee
cling
cloog
Bity aecccs
count
desaaame
Figure 6: This log-scale figure shows the extreme class imbalance of the unfiltered O1100 dataset, compared to the prefiltered IN100 dataset; certain classes which are very common in web-scraped images, such as laptops, are over- represented, while others are not represented at all. The O1100 class imbalance is produced by difference in dataset labeling strategies. VL-loss class imbalances (detected by searching for exact-match classnames in caption strings), which are present in the other web-scraped datasets in JANuS, LAION and YFCC, co-occur with comparatively low accuracy scores fine-grained classification tasks.
nateiia Peeeers en
bo seee we inei Lerss
3.5 cara
goy snsareranne
Figure ImageNet-100 samples from JANuS.
22

---

Published in Transactions on Machine Learning Research (08/2023)
Is is picuue thew offle
Rug 1186 run with bedgla toae
ate Urtrlla
N. Ihe ie ai aec
1uP iaa ze desllat st.
Bun Ihrreerrann Eeds rean!
Zuth Trny lnser!
Figure 8: OpenImages-100 samples from JANuS.
Fali Fale
12
Bleo homsmade caramber
1abe mera
Lafel Dar shong
toal honten lcoksc...
Figure 9: LAION-100 samples from JANuS
23

---

Published in Transactions on Machine Learning Research (08/2023)
Imagenet ImageNet-S: Conventional
Saption_changes
X
imagenet filkering
26
VL objective
conventional
64
Human robustness
02
ImageNet-S
X
01
*
0.5
ImageNet Base Accuracy
Figure 10: Non-linearities in ImageNet-Sketch. ImageNet-sketch performance is not linear, with only the very largest VL models showing reliable improvement over CEly trained models, when controlling for dataset size.
Imagenet v Imagenet-A: A: VL vs Conventional
caption_changes
imagenett_iileere
VL objective
conventional
Human robustness
ImagenetA
033
03
87
08
ImageNet Base Accuracy
Figure 11: ImageNet-A is learnable by all models at extremely high base accuracy. Although VL models seem to learn ImageNet-A faster than CE models, CE models reach near-parity with VL models when base accuracy gets very high.
24

---

Published in Transactions on Machine Learning Research (08/2023)
0.9
Avg. Rob.
0.8
0.7
Image Size 225px A 225px < Image Size < 385px
x 385px < Image Size
IN1000
IN100
IN100 Dogs
Label set
Figure 12:
Robustness effects of scaling input image resolution. shown that image resolution is an important factor in base model accuracy. Our meta-analysis indicates that input image resolution can have a strong positive effect on robustness as well.
Previous results from Tan & Le (2019) have
Imagenet Imagenet-R VL Conventional
29
caption_changes
X
imagenet flltering
37
objective
conventional
Human robustness
Imagenet-R
23
X
X
12
04
8.
0.5
0.
ImageNet Base Accuracy
Figure 13: VL performance on ImageNet-R outstrips base accuracy. On ImageNet-R, which is 200-class subset of ImageNet, VL models are able to achieve higher accuracy than on ImageNet itself. continues to outperform CE models on this dataset, even at very high accuracies.
25

---

Published in Transactions on Machine Learning Research (08/2023)
in1k classname
search term
good samples
total samples
avail. samples
pct. good
lion
lion
962
1000
450000
0.96
wine bottle
wine bottle
925
1000
29500
0.93
book shop
bookstore
816
984
83000
0.83
parking meter
parking meter
377
1000
9500
0.38
african elephant
african elephant
885
1000
44000
0.89
bagel
bagel
699
988
20500
0.71
tarantula
tarantula
667
981
9000
0.68
ice cream
ice cream
741
984
154500
0.75
fig
fig
517
1000
46000
0.52
shoe shop
shopping shoes
425
1000
13000
0.43
french bulldog
french bulldog hen
887
996
7500
0.89
hen
412
1000
73000
0.41
guacamole
guacamole
683
998
6500
0.68
broccoli
broccoli
679
997
19000
0.68
howler monkey
howler monkey
817
847
9000
0.96
scuba diver
scuba diver
827
1000
15000
0.83
spindle
"spindle wool, spindle -wool
311
867
867
0.36
thread"
Ihasa
Ihasa dog stoplight
719
1000
2500
0.72
traffic light
622
991
5500
0.63
lionfish
lionfish
552
897
6500
0.62
popsicle
popsicle -animal -sticks
638
943
7500
0.68
animals -insect -insects
icicle -garden -sticks -icicles
gardens -toes -label -labels
lampshade spiderweb
lampshade
446
807
6500
0.55
spiderweb
-spiderman
832
996
17500
0.84
-halloween
-pumpkin
butterfly -pleiades -nebula
-stars
lifeboat
lifeboat
572
1000
13000
0.57
cucumber
cucumber -sea -spider -beetle
730
999
26500
0.73
-flower -spiral
english springer
english springer spaniel
772
993
3500
0.78
macaw
macaw
972
1000
13500
0.97
mailbox
mailbox
900
1000
36500
0.9
peacock
peacock -butterfly
966
999
72000
0.97
bee
bumblebee OR wasp OR
686
761
110000
0.9
homet -jet -airplane
helicopter -navy -aircraft
-comic -RIAT -military
-Helicopter -Helicopters
-helicopters -aviation
Hudson -car -basketball
-sports -Transformers
cosplay -disfrazado -costume
-transformer AND flower
dungeness crab
dungeness AND crab restaurant -breakfast -lunch dinner -shack -creels -traps
474
1000
1,000
0.47
-caneery
26

---

Published in Transactions on Machine Learning Research (08/2023)
banana
banana -plant -blossom -flower -seed -seedlings -tree -spider -leaf -abstract -bay -band -festival -doll -sexy -sexiest -bread -soup -puree
793
995
65000
0.8
-smoothie -car
-plantation
-cake -cream -monkey
-pudding -z00
-republic
boxes -buying
selling
-vendor -bridge
-scone
-moon
corncob
354
1000
1000
0.35
lemon
-plant -blossom -flower -seed -seedlings
lemon
693
1000
65000
0.69
-tree -spider -leaf -abstract
-bay -band -festival -doll
-sexy -sexiest -bread -soup
-puree -smoothie -car
plantation -scent -fresh -cleaner -butterfly -grove -shots -car -sunrise -paint
graffiti origami -cake -cream
-pudding -boxes -buying -seling -vendor -scone -oon -lime
-bridge
marimba
marimba instrument orange food fruit -plant -blossom -flower -seed seedlings -tree -spider -leaf -abstract -bay -band -fstival -doll -sexy -sexiest -bread -soup -puree -smoohhie -ear -plantation -cake -cream
127
283
283
0.45
orange
744
1000
4000
0.74
-monkey -pudding -z00
-republic -boxes -buying
-selling -vendor -bridge
-scone -moon -cupcake
-cake -sales -eller -pancakes
-crepes -rep -crepe -pancake -cookie -lavoore -juice soda -pop -beach -island -cove -grove -street -drive -tea -curd -marmalade -bars -cabs chicken -cheesecake -pie
-milk
27

---

Published in Transactions on Machine Learning Research (08/2023)
bell pepper
bell pepper vegetable -plant -blossom -flower -seed seedlings -tree -spider -leaf -abstract bba -band -festival -doll -sexy -sexiest -bread -soup -puree -smoothie -car
392
505
505
0.78
-plantation -cake -cream
-monkey -pudding -z00
rrepublic -boxe buying
-selling -vendor -bridge
-scone -moon -cupcake
-cake -sales -seller pancakees
-crepes -rep -crepe -pancake
-cookie -lavoord -juice -soaa -pop -beach iisland -cove -grove -ttreet -drive -tea -curd -marmalade -bars -cabs -chicken -cheesecake -pie -milk -market -spice
espresso
espresso coffee -maker
828
1000
22000
0.83
-machine -beans -building
exterior -window mashed potato
mashed potato
635
996
10000
0.64
stingray
stingray water -dolphin shark -cruise -boat -scuba fish
600
983
2000
0.61
flagpole
fighole-lighthusse -ighthoushe -church bank -sttton
614
991
7000
0.62
teapot
teapot -tea -flower -tower -building -dome -art fashion vase -store -stores -shop -shops -Sagittarius -project365 -fountain -candle -mug -teacup -keg -vessel -amphora -urn -coffeepot umbrella
660
997
10500
0.66
umbrella
911
1000
126000
0.91
beer bottle
beer bottle -house -door
909
1003
19000
0.91
brewery -glass -cap barn -swallow -owl -bird
barn
980
1000
115000
0.98
christmas stocking
christmas stocking fireplace
317
779
779
0.41
magpie
magpie -screenshots -moth
736
983
25500
0.75
-butterfly -coprinopsis
-thieving -mushroom
mitten ram
mitten glove
800
995
1500
0.8
ram sheep -Church -window
742
1000
3000
0.74
-Window -church -school
dance -parade -festival -celebration -festivities community -fair -ewe -fox -lamb -bird -cat -dog -Dodge
28

---

Published in Transactions on Machine Learning Research (08/2023)
warthog
warthog animal -zebra
946
997
2500
0.95
cheetah -leopard -giraffe
-gazelle -hippo -rhino
donkey -armadillo -elephant
-crocodile -lion -leopard
-impala cat -monkey -bird
goose bubble
geese
474
500
69000
0.95
soap bubble -dancer -dance -fairy -tree -leaf -leaves flowers -water -toy -art abstract -museum -dog -cat butterfly -food -wine -beer chocolate -Chocolate
414
500
5000
0.83
cougar
cougar animal -wartheg
297
500
1000
0.59
-mascot -zebra -cheetah
-leopard -giraffe -gazelle
-hippo -rhino -donkey
-armadillo
-elephant
crocodile -lion -leopard
-impala -cat -monkey -bird
-lake LLake -river -River
-blonde -Blonde -woman
-girl -milf -bear -cliff -Cliffs
-cliffs -military -wallaby
-horse jett -print
daisy menu bald eagle necklace
daisy flower
500
500
52000
1
431
500
92000
0.86
bald eagle
475
500
33500
0.95
necklace jewelry -brooch
478
500
12500
0.96
-pendant -creation -earring
earrings -bracele -ring
-Engraver -bauble -anklet
chickadee
chickadee bird -Goldfinch
494
500
9000
0.99
goldfinch -robin -thrush jay -cardinal -woodpecker wren -hawk -raven -titmouse -nuthatch
stone wall flamingo gas pump vulture
"""ston wall"""
424
500
32000
0.85
flamingo bird
476
500
38500
0.95
gas station
348
500
41000
0.7
vulture bird -hawk -crow
489
500
15500
0.98
cagle
pizza
pizza pic"" -Fest -festival
305
500
1000
0.61
-summit experience -party- band -moon -parade -Parade -harvard -msaacc -montage"
29

---

Published in Transactions on Machine Learning Research (08/2023)
wallaby
wallaby
-warthog
-mascot
369
500
10000
0.74
-zebra -cheetah -leopard
-giraffe -gazelle -hippo
-rhino -donkey -armadillo
-elephant -crocodile -lion
-leopard -impala -cat
monkey -bird -koala -sports
-kangaroo -soccer -football
-ood -church -hills -stadium
-tribute -grass -rugby
apartment -car
hay
haystack field -hole -trail poster -sign
360
500
1000
0.72
grand piano
"kawai grand piano, steinway grand piano"
312
455
455
0.69
laptop
laptop
443
500
98000
0.89
dishwasher
dishwasher appliance
191
268
268
0.71
cricket
cricket -batting -sports -team
337
500
44000
0.67
-match
sea slug mongoose
nudibranch
468
500
12500
0.94
mongoose -bike -bicycle park -tree -joe -rocket military -airplane -toy -car siamese cat -bangkok -flower -snake -campaign -wat -csstmee -cosplay -festival freight
379
500
5000
0.76
siamese cat
416
500
13000
0.83
freight car vending machine bottlecap acorn
491
500
70500
0.98
.." vending machine"""
411
500
13000
0.82
bottlecap -tab
448
500
3500
0.9
acorn -woodpecker -fairy
352
500
25000
0.7
-squirrel -weevil -travel -squash strret feather boa
feather boa
135
500
2000
0.27
macaque
macaque
485
500
14500
0.97
bolete
boletus
444
500
3500
0.89
border terrier
"""border terrier"""
422
500
1500
0.84
barbell
barbells
352
500
1000
0.7
fly
housefly
398
500
1500
0.8
suspension bridge
suspension bridge
432
500
33500
0.86
jellyfish
jellyfish
477
500
46500
0.95
barbershop
barbershop -quartet -quartet-singer -singers koala
430
500
9000
0.86
koala
458
500
32500
0.92
bannister
bannister staircase
174
183
183
0.95
pillow
pillow -talk -fight caat -dog
420
500
34500
0.84
moss -sky -cloud -sky baby bib -soweer food juneo bird -finch -parroo thrush cardinal woodpecker -jay
bib
406
500
1500
0.81
junco
475
500
7000
0.95
chainlink fence soccer ball
chainlink fence ...
375
500
3500
0.75
"oscce ball -match
349
500
2500
0.7
game -milky -beach LLwees"
stupa
stupa
418
500
23500
0.84
30

---

Published in Transactions on Machine Learning Research (08/2023)
quail
quail bird -finch -sparrow thrush -cardinal -woodpecker -jay -partridge -rabbit -hawk -avocet -deer -dog -wolf coyote -gopher -eagle -vole
396
500
11000
0.79
molerat -butterfl padlock
padlock
378
500
9500
0.76
great white shark
""" great white shark"""
309
500
2000
0.62
totem pole
"""tteem pole"" wood"
383
500
1000
0.77
ant bison
ant insect bison
447
500
18000
0.89
429
500
41500
0.86
greenhouse
greenhouse
407
500
82000
0.81
Table 8: JANuS Supervision: Search Terms and Sample Quality Since many of the findings in our paper highlight the importance of both the amount and type of label noise, this table records statisties pertaining to our filtration process for the new samples in IN100. In the search term field, a symbol indicates that all samples which included that word the title, tags or description were NOT matched. Boolean OR, AND, and symbols behave they typically do.
Adding BLIP Captions to JANuS
Since we could not find human-aun numan-authored captions for ImageNet, we used BLIP Li al. (2022a) to generate descriptive captions on ImageNet-100 100. BLIP often uses word fragments to describe objects, so we used a spell checker as a simple intervention improve the quality of BLIP captions. Finally, because BLIPP' vocabulary does not include many of the specialized classes available in ImageNet, we augmented the BLIP captions with Flickr image titles, the form of text which is most commonly available for an image. We used top p=0.9, max length=40, min length=5, repetition penalty=1.1.
We repeated the process for OpenImages- 100. However, we used human-authored captions sourced from Pont-Tuset et al. (2020) instead of BLIP whenever available; around 16,000 out of the 135,000 OpenImages- 100 samples had human- authorre captions.
F Classwise Shifts
F.1 Per class accuracies for CLIP RN50 and SWSL RN50
In the supplementary attachments (2 clip__ppp____________1_111 1500mm500ppppp00 _et___errs_______eeeeeeeeeeeeee_ acc, 3_ L_sss_____i_ 1n1000_comm conf mat), we provide per-class confusion matrices on IN1000 for CLIP ResNet-50, trained on 400 Mn samples, as well as a semi weakly supervised ResNet-50 trained by Facebook on Bn samples. Yalniz et al. (2019)
In addition to classnames which are literally identical (there are two instances of the class "missile" and two instances of the class "sunglasses" in the OpenAl classnames for IN1000), find that the model struggles disambiguate short words with similar starting token strings, such "quail", "quilt" and "quill", and classes that start with common (and contextually misleading) words, such "night snake".
G Approach 2: Model details and results
Results
The complete results for approach are available part of our supplementary attachment (1_caption- net net_inlk indee wel_moood_euullet and metadata).
31

---

Published in Transactions on Machine Learning Research (08/2023)
Model details: timm
Our meta-analysis made extensive use of the popular timm Wightman (2019) computer vision library, including models from Zhang et al. (2021); Bao et al. (2022); Kolesnikov et al. (2019); Srinivas et al. (2021); Touvron et al. (2021); Xu et al. (2021); Dai et al. (2021); d' Ascoli et al. (2021); Touvron et al. (2022); Huang et al. (2016); Yu et al. (2017); Chen et al. (2017); Maaz et al. (2022); Li et al. (2022b); Xie et al. (2020); Tan & Le (2019); Wu al. (2018); Han et al. (2020); Wang et al. (2019b); Vaswani et al. (2021); Graham et al. (2021); for details about any of the timm models, please refer directly to the timm repository. In our supplementary results spreadsheet, the name field for each model is the same that model's name in the timm repository where model names have been modified between the time our evaluations took place and the time the paper was completed, we note the new names in the updated name column.
Model details: other
For non-timm models, we include a key for the terms used to describe them in the meta-analysis:
CLIP: denotes vision language pretraining using the CLIP objective
RN101: A ResNet-101 model from He et al. (2015).
RN50: A ResNet-50 model from He et al. (2015).
yfee: The yfee dataset from Thomee et al. (2016).
ce12m: The CC-12m dataset. (Changpinyo et al., 2021)
laion: the LAION dataset from Schuhmann al. (2021b).
WIT-400m: The original dataset used to train CLIP models by Radford et al. (2021).
vo-imagenet--llasnanne All samples whose caption contained an ImageNet classname were filtered out of the dataset using subset matching, as described in Sec. H.
RN50x64: A 64x scaling of the ResNet architecture according to the EfficientNet scaling rule, introduced by Radford et al. (2021).
LiT: LiT tuning as described by Zhai et al. (2021).
H Subset matching
For unsupervised web-scraped captioned datasets (such as LAION and YFCC), ground-truth class labels do not exist. Therefore. we must choose strategy to assign class labels to samples in such datasets. VL-loss models use captions as labels. There is no easy way for CE-loss models to directly use captions as labels. To facilitate this, we propose a strategy we call subset matching, modification of the "substring matching" technique proposed by Fang et al. (2022).
This strategy, illustrated detail in 14, labels samples as follows. First, construct diet of integers and "matching terms". A matching term is a string judged be good text representation of an image class, such as the string 'elephant' for image of an elephant. Our standard choice of matching terms is based on Radford et al. (2021).
If a sample caption contains a matching term, then the corresponding integer class label is applied. If the sample caption contains multiple matching terms, then we apply one of three strategies, which we label strict, multi-class (mc) and single-class (sc) matching, explained in detail in Sec. H; we use single-class matching whenever possible, since it usually performs best. the sample caption contains no matching terms for any class, then no label is applied and the image dropped from the training set. Otherwise, the caption is replaced with the corresponding integer- valued label.
A subset matching strategy is an algorithmic method for applying machine labels to images, based on caption labels. All of these methods share in common the same underlying approach, as seen in Fig. 14.
In this section, we fully define and describe some important variations on the basic subset matching strategy as de- scribed in the main paper. All of our subset matching experiments utilized one of these three strategies.
Striet: Striet subset matching means that the model only applies the label to the image the caption contains term(s) which map to exactly class.
32

---

Published in Transactions on Machine Learning Research (08/2023)
Subset Matching
Caption Processing
Image-caption Pair
Caption((Tite) (Desc.) (Tags): elephant one younger elephants exhibit remember which zoo elephant
Linear Matching
Int-Language kv pairs: 10: [elephant.pachydem] 12: [dog, canine, bbuldo).
Ngrams: (elephant, elephant one, elephant one of, one, one of, one of the, ...)
14: [200, zo0s.)
Title:
Elephant
Strict
Single-class
Multi-class
Desc: One of the younger elephants in the exhibit. @554 can't remember which zoo.
Tags: elephant
No Match
10
10, 14
Figure 14: Subset matching; overview. Subset matching is simple Jabeling strategy for unsupervised image- caption pairs. The caption is processed and converted to n-grams which are then matched against a database of terms which point to integer-label classes.
Strict subset matching was generally the most accurate method on ImageNet - we believe this is because the ImageNet dataset filtration strategy, which label selection is strongly dependent on caption contents.
It was generally less accurate method than single-class on Openlmages, where labels and caption contents are inde- pendently derived.
We find that strict matching performance tends to degrade when the pool of matching terms grow; it also tends to punish synthetic captions, which use smaller vocabulary than web-scraped or annotated captions.
Single class: In single-class subset matching, the model greedily takes the first matching term to the true class and ignores all subsequent matching terms.
As general matter, found that single-class matching struck the best balance between dataset utilization and accuracy, and we used this method for most of our experiments.
Multi class: In multi-class subset matching, we match up to 25 classes per sample (if we see multiple terms for single class, we ignore those additional terms, and we do not attempt to rank classes by frequency).
The cross-entrops loss of the model is then given by the sum of the loss on each class; in other words, we reward the model for applying a high probability on each label assigned to the sample and for applying low probability to each label which was not assigned the sample.
This approach, while intriguing, was challenging because only had one ground-truth label for each image; there- fore, multi-class matching was always less accurate than single-class matching in direct comparison.
Since our cross-entropy model used a softmax loss, we found that model error tended to be high as the number of matched classes grew. We also found empirically that images which actually required multiple labels were not partic- ularly common in our dataset. Perhaps for these reasons, this approach performed worse than single-class matching in most experiments.
Additional term definitions.
33

---

labels which match ground truth labels, divided by the total number of samples in the Published in Transactions on Machine Learning Research (08/2023)
Label accuracy. On datasets for which supervised ground truth labels exist, we report label accuracy (Label acc.) the count of dataset.
Dataset utilization. Dataset utilization (Ds. Util) of a model on dataset is the ratio of correctly labeled samples to total number of samples (including correct, incorrect and unlabeled samples). We use this metric to judge how useful labeling strategy is; ground truth labels have a utilization of 100%; automated labeling methods gives typically significantly less utilization.
Approach Results
Please see Tab. for the results for models trained using approach (JANuS). Baseline result for each constituent dataset is in bold.
RESULTS KEY
imbal: Class-imbalanced dataset
384-res: Trained at 384 image resolution
RN50x4: Used RN50x4 backbone
RN26: Used RN26 backbone
int: Trained using CE-loss
VL: Trained using VL-loss
ipeg10: JPEG compression of images, strength 10
cliplabel: Trained using noisy integer labels generated by CLIP VIT-L-144 model
swinlabel: Trained using noisy integer labels generated by Swin transformer
rtd: VL model trained using noisy captions (title, tag, description from Flickr where available, alt-text where not available)
lags: VL model trained using tags only
title: VL model trained using title only
sbm: Samples selected using subset matching
size: Sample size was size-controlled to be identical to supervised dataset size
grcaps: VL model trained on ground-trute ttunh captions generated from human labels
tokscramble: VL model trained with the order of tokens randomly scrambled during training and inference
tokstrip: VL model trained with all tokens not used for inference stripped during training
blipcap: Captions generated using BLIP captioning model
ofa: Captions generated using an OFA captioning model
annotcap: Human descriptive annotations for images (available only for a subset of 01100)
classname-only: Captions are classname and nothing else
classbal: Corrected for class imbalance prior training
34

---

Published in Transactions on Machine Learning Research (08/2023)
Model Name
IN100-Val IN100-V2
IN100-S
IN100-R
IN100-A
Avg, Rob:
Eff. Rob.
Label Acc.
DS Util.
Data Budgeet:
1/8x
in100K 110001//8000 i8tt
0.618
0.485
0.139
0.189
0.104
0.229
0.371
1-100 VL-gicaps
0.577
0.461
0.17
0.196
0.104
0.233
0.402
Budget:
(bal)
In100-int
0.87
0.791
0.373
0.378
0.153
0.424
0.487
1111000nt-88--ree
0.841
0.768
0.337
0.343
0,177
0.406
0.483
in100 int RN26
0.81
0.736
0.377
0.363
0.135
0.403
0.498
in100-int-Re 5004-
0.874
0.805
0.336
0.369
0.19
0.425
0.486
100--ittVVT----16
0.124
0.204
0.359
in100-int-DeTT-S-16
0.756
0.643
0.164
0.222
0.137
0.292
0,386
in100-in-GGCVViI-S---
0.803
0.702
0.378
0.356
0,143
0.395
0.492
in100-int-jpeg11
0.809
0.728
0.341
0.345
0.131
0.386
0.478
in100 cltpcail
0.813
0.717
0.278
0.328
0.127
0.363
0.446
0.9
in100.inttsieesssmmtt
in100-int-shm-ttt
0.754
0.674
0.285
0.331
0.123
0.353
0,468
0.89
0.72
in100-intsson-ttas
0.723
0.636
0.251
0.297
0,109
0.323
0.447
0.87
0.58
in in100-int-bmm-titte
0.686
0.603
0.237
0,301
0,107
0.312
0.455
0.94
0.49
in (n100--i..ao 100-VI-ggsspp
0.849
0.768
0.37
0.373
0.17
0.421
0.495
0.765
0.372
0.399
0.162
0.425
0,507
in100-VL VL-j---
0.75
0.682
0.311
0.352
0.144
0.372
0.496
in100-VL-t0 VL-VVL-
0.587
0.487
0.162
0.173
0.085
0.227
0.386
0.89
0.72
in100 VL-ud-tokkstrg
0.585
0.475
0.145
0.19
0.081
0.223
0.381
0.89
0.72
in100-V in100-VL-blipcag VL-blipcup
0.405
0.351
0.138
0.165
0.07
0.181
0.447
0.61
0.28
0.236
0.218
0.122
0.05
0.521
Data Budget:
(imbal)
100-int
0.667
0.595
0.316
0.399
0.156
0.367
0.549
oi o0-0niint-llssb
0.812
0.734
0.39
0.399
0.167
0.423
0.520
]
oi 1i100-int-liiplael
0.631
0.553
0.273
0.343
0.134
0.326
0,516
0.9
oi100-int-sbm-ttu
0.369
0.304
0.109
0.2
0.104
0.179
0.486
0.48
001000-VL.. VL--aaa
0.644
0.423
0.177
oi100-VL<td
0.26
0.22
0.065
0.121
0.066
0.454
0.53
0.11
o1100-VL-bil
0.343
0.291
0.09
0.174
0.055
0,152
0.443
0.46
0.14
od os100 200-VL- blipcpp
0.298
0.28
0.095
0.151
0.065
0.148
0.495
0.42
0.12
Data Budget:
0:10000-i100-innt res
0.904
0.829
0.489
0.515
0.213
0.512
0.566
0.442
0.455
0.241
10000 100-int-ViTS.
0.785
0.667
0.211
0.277
0.146
0.325
0.414
in 100 1000-nt-RNN
0.858
0.771
0.463
0.465
0.204
0.476
0.555
in
0.903
0.838
0.458
0.478
0.229
0.501
0.555
Data Budget:
im1000-laion10--1
0.901
0.827
0.614
0.636
0.201
0.57
0.633
NIA
N/A
(0-laiio 100-int
0.314
0.379
in100+taion100-4 VO-VLLud
0.529
0.289
0.299
0.283
N/A
Data Budget:
10x
JANuS-int
0.908
0.863
0.678
0.731
0.35
0.656
0.722
N/A
N/A
JANuS 384-384-res
0.868
0.792
0.559
0.622
0.355
0.582
0.671
N/A
N/A
JANUS RN5004
0.909
0.866
0.66
0.718
0.387
0.658
0.724
NIA
N/A
JANuS-int: VIT-S-16
D.564
0.908
0.863
0.678
0.731
0.349
0.655
0.721
N/A
N/A
JANuS-int-tt-sbm--..1
0.871
0.817
0.625
0.659
0.276
0.594
0.682
N/A
N/A
JANuS: RI.S.VV..aa Vl-----td---dd.mmm
0.846
0.757
0.447
0.506
0.204
0.478
D.566
N/A
N/A
JANuS-VL-offal.Im
0.67
0.587
0.392
0.453
0.147
0.395
0.589
N/A
N/A
Data Budget: 20xx
0.927
0.877
0.78
0.449
0.702
0.757 0.656
NIA
N/A
JANuS-ylle 2.4m cliplabels ViT.S.
Table 9: Approach 1; Full results. Results for models trained from scratch using Approach 1 (JANuS).
35

---

Published in Transactions on Machine Learning Research (08/2023)
Class Frequency Count
Cuust
500
dan por Eggarr sxtarael
de- aerto bricking
Melartta michers
to slock
mode
Figure 15: Class frequency count using subset matching on ImageNet-100. 100.
Class Frequercy cy Count
620
gers
bo farme
Label
Figure 16: Class frequency count using subset matching on YFCC-100.
J Class Frequency Counts for IN100 subset matching distributions, openai labels, mc matching
The figures referenced below depict the distribution of noisy, web-scraped labels with respect to true labels for each of the constituent datasets in JANuS.
ImageNet-100: 15. YFCC-100: 16. LAION-100; 17.
Cless Frequercy Count
42c
Court
430
234
onoet mode
Safte Miterstap
pople
parp
Label
Figure 17: Class frequency count using subset matching on LAION-100.
36

---

Published in Transactions on Machine Learning Research (08/2023)
Per dass accurazy of subset matahing
de on
wilh
mecbert
Figure 18: Per class accuracy when using subset matching on IN-100.
Per dass subset matching
M
mistetee eether
moderr
bolppe
Lobel
Figure 19: Per class accuracy when using subset matching on OI-100.
K
Per Class Accuracy for Subset Matching, openai classnames, sc
We report the accuracy of the subset-matched labels, compared to ground-truth labels, for the following constituent datasets in JANuS:
ImageNet-100: 18. OpenImages-100: 19.
L JANuS Spreadsheet Column Explanations
JANuS contains many different kinds of metadata, and the meaning of some of the column labels used may not be immediately apparent to the reader.
We do not provide explanations for metadata columns which are explained in one of the original dataset descriptions; for those, we recommend referring to the original authors of the datasets. (Deng et al., 2009; Fang et al., 2022: Schuhmann et al., 2021a; Thomee et al., 2016; Kuznetsova al., 2020)
BLIPCaption refers captions generated by us using a BLIP captioning model. BLIPTitle captions are combina- tion of the BLIP caption and the title field of flickr captions. Li et al. (2022a)
FlickrCaption refers to captions sourced from flickr.
onnot_caption refers to Openlmages captions that were authored by human image annotators. prose_caption com- bines BLIP and annotator captions, favoring the latter when available.
clip_ie idx are ImageNet labels chosen by zero-shot CLIP ViT-L model from OpenAL.
idx labels refer to labels generated using various subset-matching strategies.
37

---

Published in Transactions on Machine Learning Research (08/2023)
me is multiclass, is single class, strict is strict. Ours, default, openai refer to the three different sets of class labels we experimented with throughout this paper.
38


Reviews:
[{"review_id": "2tkr4842sq", "reviewer": "Reviewer_DK2J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- All in all, the paper is well written and easy to follow.\n- The introduction of JANuS, provides a unique resource for studying robustness in image classification. Such a dataset can significantly facilitate future research. \n- The paper demonstrates the possibility of training highly robust models using conventional cross-entropy loss, even when data and model sizes are limited.\n- The authors of the paper provide comprehensive experimental analysis. This large-scale evaluation enhances the reliability and generalizability of the findings.\n- The paper outlines practical heuristics for improving distributional robustness when faced with limited data budgets.\n\nWeaknesses: \n- The paper might have limited algorithmic novelty. \n- Since the JANuS dataset is derived from several existing datasets based on automated selection criteria like subset matching as described in the paper. This can cause issues like dataset bias and overfitting. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of training robust deep learning models under distribution shifts, even with limited data. The authors introduce a new dataset called JANuS, which includes images, labels, and captions, and use it to investigate factors contributing to robustness in image classification.\nThe authors demonstrate that it is possible to train highly robust and accurate models using conventional cross-entropy loss, even with limited data and model sizes. They conduct a comprehensive analysis  and show that modest scaling of models and data can lead to robust models on large and fine-grained label sets.\nThe paper's contributions include showing the feasibility of training robust models with limited data and model sizes, introducing the JANuS dataset, conducting a large-scale analysis of image classification models, providing heuristics for improving distributional robustness with limited data, and releasing code, dataset, and results for reproducibility.\nOverall, the paper provides valuable insights into training robust deep learning models under limited data conditions and offers resources for future research in distributional robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be great if the authors of the paper can describe further about the proposed JANuS dataset, and provide some metrics for us to further understand the dataset. I see that additional descriptions are available in the appendix, but it might be beneficial to include some of the details in the main text, as I think the dataset is a major contribution of the paper.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rHW8Z4gTYm", "reviewer": "Reviewer_FEFA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The empirical studies are thorough, especially the ablation studies. \n- The paper is clearly written.\n\n# Weaknesses\n\n- It seems that the large and fine-grained label sets contribute to the success of this paper. What is the underlying reason? Has the model witnessed enough variations?\n\n- We know CLIP can work on noisy text supervision. Besides, CLIP is not targeted for image classification---it can serve as general feature extractors, as in diffusion models and mini-gpt4. So, is the comparison between your method on CLIP fair?\n\n- What if we fine-tune CLIP to handle the problem you mentioned? I think CLIP can serve as a good warmup.\n\n- Overall, the conclusion that large and fine-grained label sets yield better robustness is unsurprising. And honestly, we cannot regard 2.4M samples as a limited budget. So I question the significance of this paper.\n\n# Minor \n- what is the SWSL in table 1?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically investigates the factors affecting image classifiers' robustness against natural distribution shifts. By evaluating on various datasets, architectures, and training data sizes, the authors found that we can train robust learners in a domain where data is limited. It seems that the key to success is the large and fine-grained label sets. \n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the issues in the Weaknesses part.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1bXKPkyBag", "reviewer": "Reviewer_bBmd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n#### 1. **Topic of great interest**: \nThe paper addresses a highly relevant and actively explored research area. Understanding the key factors that contribute to robustness in modern deep learning practices is of great interest to the wide TMLR audience.\n\n#### 2. **JANuS could be a useful dataset**: \nThe release of the JANuS dataset is a valuable contribution of this work. The dataset provides a standardized, relatively clean dataset with access to labels and captions for all samples. This will facilitate future studies aiming to analyze the differences between vision-language models and image-only models. I also agree with the authors that the coarser granularity of the label space of JANuS enables the development of high-performing models at a lower training cost, which could potentially democratize research in this domain.\n\n## Weaknesses\n\n#### 1. (Major) **Cumbersome presentation of results and lack of experimental details**: \nThe primary drawback of this paper lies in the cluttered and poorly motivated presentation of results in Section 6. The experimental settings for each \"ablated\" factor of variation are unclear, and the rationale behind the experiments is not adequately explained. The paper fails to deliver on its promise of conducting a controlled and unified benchmarking of distinct approaches, as the presented results mix multiple factors of variation without proper unity. The mix usage of models pre-trained on different sources, the inconsistent treatment of sample numbers, or free use of different architectures further exacerbates the lack of clarity of the section. Overall, the narrative is extremely challenging to follow, as the experiments are not properly described, making it difficult for readers to draw meaningful conclusions.\n\nAs an example, the paragraph discussing the **Model inference comparison** should focus on the effect of data scale. However, the results also delve into a comparison between VL-loss and CE-loss, along with an experiment involving different ways to filter examples and a comment about the impact of using various pre-training datasets (with missing details). While these experiments might not be inherently wrong, their mixed presentation and possibly inconsistent execution render them incomprehensible for readers seeking meaningful insights.\n\nIn general, the entire Section 6 suffers from similar issues and is ridden with missing experimental details. Many times in the text the authors comment on findings for which no supporting table or figure is provided. Most importantly, the whole section **E. Model Training Details** is missing from the appendix This is problematic, as the lack of a consistent experimental setup, along with the absence of supporting tables and figures in the missing further hinders the reader's understanding and compromises the validity of the findings.\n\n#### 2. **Unclear conclusions**: \nThe difficulties posed by the mixed and convoluted experiments make it challenging to derive clear conclusions from this work. The guidelines presented in Section 7 lack a solid experimental foundation, as they are intertwined with numerous confounding factors. Moreover, while the JANuS dataset holds potential value, its utilization is unclear, as the paper fails to clearly indicate when it is used and when it is not, undermining its potential contribution to the readers.\n\n#### 3. (Minor) **Formatting errors**: \nSeveral formatting errors are present throughout the paper: The usage of `\\citep` and `\\citet` macros in LaTeX is frequently incorrect. Additionally, the decimal numbers in the tables inconsistently exhibit a leading zero, and the sudden transition in the appendix from a single-column format to a multi-column format is inconsistent and requires attention.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new study focused on identifying the primary design features that contribute to distributional robustness in contemporary deep learning practices. By exploring various dimensions of variation, including model size, architecture, data diversity, and training protocols, the authors aim to uncover the crucial factors that underpin robustness. To facilitate controlled scaling experiments, the paper introduces JANuS, a novel training dataset designed to elucidate the impact of different data scales on robustness in modern deep learning. JANuS is a compilation of four publicly available training datasets that have undergone major cleaning and unification processes. It features a single label and caption space, ensuring compatibility with the ImageNet-100 dataset and its shifted variants, which serve as the main basis for evaluations conducted in this study. At the end, the paper states some guidelines for practitioners seeking to enhance robustness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To recommend acceptance, I would require a substantial overhaul of Section 6 in the revised version of the manuscript. It is crucial that this section is divided into different subsections to improve clarity and organization. Each experiment should focus on studying a single factor of variation in isolation, while keeping all other variables fixed. While some degree of stratification, such as distinguishing between VL-loss and CE-loss in certain cases, may be acceptable, it should be properly justified. The key aspect is that each experiment should convey a clear and unique message to the reader.\n\nTo enhance the comprehensibility of the findings, it is paramount that every statement is supported by relevant tables or figures. The most significant and important findings should be presented directly in the main text, while any supplementary or less central information can be deferred to the appendix. It is essential to minimize the need for constant referrals to the appendix, ensuring that the main paper provides a comprehensive overview. Care should be taken to avoid missing details, and all relevant information should be included in the text. I expect to see evidence that each of the guidelines in Section 7 are properly supported by a clean experiment.\n\nIf these changes are not implemented to a degree that makes the presented findings reliable, I will regrettably recommend rejection of the manuscript.", "broader_impact_concerns": "I don't see any clear negative societal impacts directly stemming from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summarising reviewers final comments, post revision reviewers' concerns regarding the lack of evidence for the results have been significantly reduced. The paper reads better, although it still could be improved. 


Decision: Accept (poster)