LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Authors: Fanfei Li
Thomas Klein
Wieland Brendel
Robert Geirhos
Roland S. Zimmermann
Venue: ICML 2025 poster (2025)
Keywords: OOD
representation learning
benchmark
model evaluation
vision
classification
psychophysics
Primary area: deep_learning->robustness
TL;DR: We introduce a new OOD robustness benchmark for web-scale models along with human performance results.

Abstract: Out-of-distribution (OOD) robustness is a desired property of computer vision models. Improving model robustness requires high-quality signals from robustness benchmarks to quantify progress. While various benchmark datasets such as ImageNet-C were proposed in the ImageNet era, most ImageNet-C corruption types are no longer OOD relative to today's large, web-scraped datasets, which already contain common corruptions such as blur or JPEG compression artifacts. Consequently, these benchmarks are no longer well-suited for evaluating OOD robustness in the era of web-scale datasets. Indeed, recent models show saturating scores on ImageNet-era OOD benchmarks, indicating that it is unclear whether models trained on web-scale datasets truly become better at OOD generalization or whether they have simply been exposed to the test distortions during training. To address this, we introduce LAION-C as a benchmark alternative for ImageNet-C. LAION-C consists of six novel distortion types specifically designed to be OOD, even for web-scale datasets such as LAION. In a comprehensive evaluation of state-of-the-art models, we find that the LAION-C dataset poses significant challenges to contemporary models, including MLLMs such as Gemini and GPT-4o. We additionally conducted a psychophysical experiment to evaluate the difficulty of our corruptions for human observers, enabling a comparison of models to lab-quality human robustness data. We observe a paradigm shift in OOD generalization: from humans outperforming models, to the best models now matching or outperforming the best human observers.

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Fanfei Li"
Thomas Klein'
Wieland Brendel 123
Robert Geirhos 14
Roland S. Zimmermann"
Abstract
1. Introduction
Out-of difrriiiiiiittiionn (OOD) robustness is de- sired property of computer vision models. Im- proving model robustness requires high-quality signals from robustness benchmarks to quan- tify progress. While various benchmark datasets such ImageNet-C: were proposed in the Im- ageNet era, most ImageNet-C corruption types are no longer OOD relative to today's 's large, web- scraped datasets, which already contain common corruptions such as blur or JPEG compression artifacts. Consequently, these benchmarks are no longer well-suited for evaluating OOD ro- bustness in the era of web-scale datasets. In- deed, recent models show saturating scores on ImageNet-era OOD benchmarks, indicating that it is unclear whether models trained on web-scale datasets truly become better at OOD generaliza- tion or whether they have simply been exposed to the test distortions during training. To address this, we introduce LAION-C as benchmark al- ternative for ImageNet-C. LAION-C consists of six novel distortion types specifically designed to be OOD, even for web-scale datasets such as LAION. In a comprehensive evaluation of state- of-the-art models, we find that the LAION-C dataset poses significant challenges to contempo- rary models, including MLLMs such Gemini and GPT-4o. We additionally conducted psy- chophysical experiment to evaluate the difficulty of our corruptions for human observers, enabling a comparison of models to lab-quality human ro- bustness data. We observe a paradigm shift in OOD generalization: from humans outperform- ing models, to the best models now matching or outperforming the best human observers.
In recent years, large-scale vision models such as vision transformers (Dosovitskiy et al., 2021) and ConvNeXt (Liu et al., 2022), trained on expansive web-scale datasets like LAAONN (Schuhmann et al., 2022), have pushed the bound- aries of performance on standard benchmarks. However, the continued advancement and reliable evaluation of these models depends the availability of datasets that effec- tively challenge model robustness and generalization capa- bilities.
In the era of training models on curated datasets like ImageNet (Russakovsky al., 2015), creating OOD- benchmarks was relatively straight-forward: By introduc- ing visual corruptions that were absent from the training set, such blur and noise, researchers could evaluate the robustness of their models in a controlled manner. If a model performs well on a corruption it has never seen, it can be said to be robust to this corruption. For example, ImageNet-C (Hendrycks & Dietterich, 2019), which intro- duces different parametric corruptions the ImageNet val- idation set, has long stood as the de facto standard for OOD evaluation. Models that were trained on the (uncorrupted) ImageNet images must robustly generalize in order to per- form well on ImageNet-C.
With the shift towards training models on vast, largely un- filtered image datasets, much less clear how to obtain test images that are truly OOD, Adding noise and blur to images can longer be considered distribution shift, be- cause such images are already present in the training set, as demonstrated in Fig, 1. Models trained on LAION have seen the types of corruption in ImageNet-C, and are likely exposed to all realistic corruptions. Therefore, the fact that models like CLIP (Radford et al., 2021) exhibit much bet- ter performance on classic OOD datasets than ImageNet- trained models might not be indication of true OOD ro- bustness, but rather consequence of smaller train-test gap. For distribution shifts defined by the style of an image, recent work empirically shows that such datasets are indeed not OOD, but overlap with LAION-400M (Mayilvahanan et al., 2023; 2024). Hence, to measure the OOD robustness of modern models, a dataset containing truly new image corruptions is needed-even if this means that the corrup-
*Supervised this work. 'Max Planck Institute for Intelli- gent Systems, Tübingen, Germany FELLIS Institute Tübingen Tübingen Al Center 'Google DeepMind. Correspondence Fanfei Li first. @ cuebingen.mpg.de>
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LAION-C: An Out-of-Distribution Benchmark for Web-Seale Vision Models
tions must be highly artificial.
P(X) changes, while P(Y remains the same (as op- posed concept drift or label shifts, commonly referred to as "semantic" shifts (Yang et al., 2024)). Our dataset in- troduces six carefully designed, fully synthetic distortions that models have not encountered during training. These distortions are designed to be OOD even for web-scale datasets (as supported by quantitative evidence presented later). Hence, models truly need to generalize beyond their training distributions to perform well on this benchmark which we term LAION-C.
Previous work (e.g., Hendrycks & Dietterich, 2019) found that OOD generalization is not trivial to achieve: Many vision models do indeed struggle with OOD datasets like ImageNet-C, even if they perform well on ImageNet itself. Hence, these types of unfamiliar inputs are crucial for evaluating the robustness of machine learning models since they are indicative of performance on unexpected in- put; a challenge that many deployed models face.
Given the importance of OOD generalization in practice, there is a pressing need for a new benchmark that more effectively evaluates the OOD robustness of state-of-the- art foundation models: an OOD dataset for the era of web- scale vision models. Our contributions are as follows:
The core idea behind our distortions is cre- ate highly synthetic, artificial corruptions that do not natu- rally appear even in large-scale datasets and are therefore OOD even for modern vision models. To achieve this goal, we intentionally disrupt visual consistency and perceptual cues that models might rely on for image classification. We began with a diverse pool of potential distortions, each de- signed to target different aspects of visual processing: tex- ture handling (Geirhos et al., 2019), color perception, or edge detection. Initial testing revealed that while contem- porary models are adept at navigating simpler distortions, their performance varies significantly with more complex challenges. This insight prompted us to carefully choose distortions that provide a sufficient challenge the models while also offering varied levels of perceptual difficulty for humans.
Distortions
1. Given that existing OOD datasets are often no longer OOD for models trained on web-scale datasets, we in- troduce LAION-C, a novel benchmark dataset with six manually designed corruption types and 16 superclasses to evaluate the robustness of web-scale vision models. This dataset serves proxy for unseen challenges, al- lowing us to probe the limits of current models' robust- ness in a controlled but challenging environment.
2. We conduct a comprehensive performance analysis of various models on LAION-C and report a robust human OOD generalization baseline obtained through psy- chophysical experiments with 19 participants, collect- ing 1,,400 trials in highly controlled laboratory envi- ronment.
Our final distortions are selected to fulfill two key desider- ata: (1) they are exotic enough to have a low occurrence probability even in web-scale datasets, and (2) they test fea- ture extraction capabilities relevant to robust object recog- nition. For instance, solving the Stickers and Mosaic dis- tortions requires a model holistically integrate the image, rather than being misled by the local image cues introduced by sub- sub-images- a scenario deep neural networks notori- ously struggle with (Geirhos et al., 2019). The Glitch and Vertical Lines distortions are among the most exotic and globally disruptive transformations we could construct, as they effectively destroy the texture cues that many models rely on (Geirhos et al., 2019). The Geometric Shapes dis- tortion tests amodal completion, fundamental aspect of human visual processing observable even in infants (Kell- man & Spelke, 1983; Nanay, 2018), and also alters the image's color to, as we do not primarily depend on color for object recognition (Biederman & Ju, 1988; Tanaka & Presnell, 1999). Lastly, the Luminance Checkerboard distortion tests model's 's ability to adapt to local lighting conditions, an essential ability of the human visual system (Carandini & Heeger, 2012). This way, LAION-C complements exist- ing benchmarks by including images that challenge human perception, instead of limiting ourselves to visual domains in which humans excel.
3.
The resulting data serves as an OOD benchmark for cur- rent and future models, enabling not only assessment of their generalization ability on truly OOD data but also providing insights into the discrepancies between hu- man and machine perception, observing a paradigm shift in OOD generalization: from humans outperform- ing models to the best models now matching or outper- forming the best human observers.
A detailed related work section can be found in Appx. A.I.
2. Methods
crabbuttoo--smmtttinn humans are robust
2.1. Constructing New OOD Distortions
As described in the introduction and depicted in Fig. 1, ImageNet-C does no longer qualify as out-of-distribution (OOD) for models trained on large-scale datasets and, therefore, can no longer be employed for such test- ing. Given the limitations of existing benchmarks like ImageNet-C, we develop a novel dataset specifically de- signed to challenge these foundation models more rigor- ously. We do this by introducing a covariate shift, where
2

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
ImageNet-C
LAION 400M
Noise
Blur
Weather
Digital
Figure I. ImageNet-CC corruptions are not out-of-distribution (OOD) for web-scale datasets like LAION-400M. Exemplary cor- rupted images from ImageNet-C (left) similar LAION-400M samples (right). Each shows example corruptions and dataset images for one ImageNet-C corruption category (Noise, Blur, Weather, Digital). The presence of these distortions in web-scale datasets indicates the need for OOD benchmark in the of web-scale vision models.
As AI systems are increasingly deployed in complex and high-stakes domains, it is crucial that benchmarks evolve to robustly test these systems' generalization capabilities. Following ImageNet-C, each distortion consists of five dif- ferent intensity levels. The distortions capture a range of vi- sual challenges, described below and illustrated in Fig. 2.
Geometric Shapes: The original image is overlaid with overlapping geometric figures such as squares, circles, and stars. This visual clutter introduces local noise that obscures the main object, like the Kaleidoscope corrup- tion from (Kaufmann et al., 2019).
Stickers:
The original image is augmented with assorted image patches. This addition of visual elements masks features of the primary object.
The original image is broken down into smaller tiles, each replaced by chromatically similar picture. This patchwork creates a mosaic effect that disrupts edges and textures while introducing contextually irrel- evant information.
Mosaie:
The original image di- vided into grid, with the luminance of each cell altered in checkerboard pattern. The stark luminance con- trast between adjacent tiles and artificial grid boundaries makes this distortion challenging.
Luminance Checkerboard:
The original image undergoes an artistic digi- tal corruption with horizontal lines overlaying shifted im- age segments and color channel shifts. This dislocates the global contextual structure of the image. While the concept of such glitchy images has been explored in lier work (Kaufmann et al., 2019), our transformation in- troduces more intense corruption.
Glitched:
We intend to build a challenging dataset that has the po- tential to guide the future development of vision models. Our dataset incorporates these tougher and less common distortions simulate the difficulty of OOD scenarios that models might encounter in real-world applications. In line with established psychophysical experimental paradigms, we calibrate the intensity levels of our distortions to range from near-pers near-perfect to chance-level difficulties. We tune these levels such that either humans or contemporary vision model (ViT-B) achieve chance performance on the
The original image is deconstructed into bent vertical line segments. This distortion retains the original colors but strips away local information, disrupt- ing the contours and edges of the image and introducing visual discontinuity.
Vertical Lines:
3

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Geometric Shapes
Luminance Checkerboard
Mosaic
Glitched
Vertical Lines
Stickers
Intensity 1
Intensity 2
Intensity 3
Intensity 4
Intensity 5
Figure 2. LAION-C distortions, intended to be OOD even for web-scale datasets. This figure illustrates the six LAION-C distortions at five intensity levels. Following the standard experimental paradigm from psychophysics, our dataset spans from near-perfect to chance-level difficultics, thoroughly testing models and leaving room for future model improvements. Best viewed sereen.
highest intensity level, i.e. model is expected perform well the hardest levels. The other intensity levels are chosen so that we observe gradual decline in accu- racy, ensuring that the distortions are sufficiently challeng- ing.
types, resulting in 130k total images. This data size lection allows ensure that 3% difference in the per- formance between models is statistically significant. Addi- tionally, we manually filter the dataset to ensure that none of the images in one superclass contain objects from an- other class or require specific cultural knowledge for clas- sification. This process helps to avoid ambiguous ground- truth labels.
These distortions are then applied a carefully curated subset of images from the ImageNet validation dataset. To contextualize model performance, we intend to compare it to human performance. As human evaluations datasets with hundreds of classes cannot be scaled to sufficiently many participants, we follow previous work (Geirhos et al., 2018) and simplify the task to a 16-class classification problem. We extract 285 ImageNet-classes to form 16 su- perclasses, namely ball, bird, boat, bottle, butterfly, car & truck, cat, chair, dog, fish, fruit, instrument, primate, snake, timekeeping, and tool. For robust statistical analysis, our dataset comprises 273 images for each superclass, each of which is corrupted at 5 intensity levels for all 6 distortion
2.2. Measuring Model Performance
We use the generated datasets to evaluate the performance of a suite of 58 vision models. Our selection includes mod- els trained ImageNet- Ik, such Vision Transformers (VTT) (Doso- vitskiy et al., 2021), ConvNeXt (Liu et al., 2022), and EVA (Fang et al., 2023; 2024). For comparison, we also evaluate the performance smaller-scale model families such as ResNet (He al., 2016) and MobileNet (Howard,
large-scale web datasets and fine-tuned

---

LAION-C: An Out-of-Distribution Benchmark for Web-Seale Vision Models
2017) and large-scale models trained only on ImageNet- Ik. Additionally, we also evaluate GPT--4 (OpenAI, 2024) and Gemini 1.5 Pro (Team et al., 2024) on representative sub- set of LAION images. See Tab. 11 for a complete list of all models we evaluate. To address the imbalance caused by distinct numbers of subclasses within each superclass, we compute the average probability values across subclasses for each superclass. a method first suggested by (Geirhos et al., 2018). This adjustment mitigates biases introduced by the varying subclass distributions, ensuring more ac- curate model performance evaluation.
detailed explanation of the metric.
3. Results
3.1. How OOD is LAION-C?
Next, we empirically evaluate whether our LAION-C dataset is indeed OOD relative to the large-scale image datasets used to train modern vision models. Rigorously quantifying how OOD a test dataset is with respect to some training dataset requires precise definition of the test and training domain (Mayilvahanan et al., 2024). As the dis- tribution shifts introduced by the distortions of our pro- posed LAION-C and ImageNet-C are fuzzy in nature, we use three tools to compare the OOD-ness of our proposed dataset to the OOD-ness of ImageNet-C. First, we use a qualitative assessment. By searching for the name and re- lated concepts of ImageNet-C corruptions, we easily find LAION samples with visual distortions akin to those of ImageNet-C samples (see Fig. 1).
2.3. Collecting Human Performance via Lab Experiments
To explore the discrepancies between human and machine perception, we design a psychophysical experiment to gather human classification data on the augmented images. This experiment builds on previous paradigms (Geirhos et al.. 2018; 2021) to ensure consistency and comparabil- ity. In the experiment, 19 human subjects briefly pre- sented with distorted image and are asked to classify it into one of 16 classes, reminiscent of how DNN might be evaluated on classification task (in contrast to e.g. open response paradigms, where participants could give arbi- trary textual responses). Participants were recruited from the university student body, and screened for normal or corrected-to-normall vision and absence of color blindness. The experiment was conducted in controlled dark envi- ronment using 22" VIEWPixx 3D light LCD monitor, with stimuli presented centrally at fixed viewing distance to ensure foveal presentation. The classification task was divided into two warm-up blocks and ten main experiment blocks, with each block containing images from 16 super- classes. Participants were given 2.5 to view each image, followed by response window to classify the image by clicking a set of icons. To motivate high performance, a monetary bonus was awarded for surpassing fixed, pre- determined performance thresholds for each block. Further methodological details are provided in Appx. A.2.
Second, we use the difficulty of a test dataset (measured by the performance that models trained on a reference dataset yield on the test dataset) as a proxy for how much the test dataset differs from the reference dataset. Here, the reason- ing is that if a test dataset can be solved almost perfectly by model, it means that either the model has great gen- eralization skills or the test dataset is not strictly OOD. If, at the same time, another dataset has much greater diffi- culty according to the same models, the second dataset is likely more OOD than the first For the sake of compa- rability, we here use version of ImageNet-C restricted to the same 16 superclasses that were used for LAION- C, where implemented the ImageNet-C augmentations through the code by Michaclis et al. (2019). Indeed, com- parison of the performance achieved by our suite of mod- els (see Fig. 3) suggests that LAION-C is more OOD to LAION than ImageNet-C is.
100-
Accraty I (10))))
2.4. Quantifying Human-Machine Alignment via Error Consistency
DanseNet
ResNet
WideNet
EfficientNet
To quantify the alignment between human and machine vi- sual perception, we adopt the error consistency metric pro- posed in Geirhos et al. (2020b). Error consistency, denoted as i E [-1, measures the degree of agreement between the classification mistakes of two observers. In brief, x takes on a value of 1.0 if two observers are perfectly con- sistent, i.e. if they make classification mistakes on exactly the same images. Two independent binomial observers that agree no more than expected by chance will result in a k of O, while two maximally inconsistent observers will have i. of -1. See Appx. A.3 or Geirhos et al. (2020b) for
70
X
ConvWext
VIT
Gemini-1. 1.5rr
60
GPT-40*
EVA
50
80
100
Accuracy on (96)
Figure 3. Performance Divergence of Models on LAION-C and ImageNet-C 16 class. Evaluating models on the 16-class versions ImageNet-C and LAION-C produces a plateaued per- formance on ImageNet-C, while LAION-C still yields a high vari- ance across models.
5

---

LAION-C: An Out-of-Distribution Benchmark for Web-Seale Vision Models
Third, we use the FID (Heusel et al., 2017; Kynkäänniemi al., 2022) quantify the difference between LAION and ImageNet-C and LAION-C, respectively, Specifically, we employ a CLIP-trained ViT-B as feature encoder and use the implementation by Parmar et al. (2022) to calculate FID-scores. In line with the previous evidence, we find a FID of 70 between LAION and LAION-C, which is sub- stantially higher than that between LAION and ImageNet- C (== 40). This means that features of LAION are closer to those of ImageNet-C than those of LAION-C, again high- lighting the greater OOD--ess of LAION-C. In summary, we have presented three different kinds of evidence sug- gesting that LAION-C is more OOD than ImageNet-C to LAION.
60%, indicating that current models are increasingly adept at handling the distortions in ImageNet-C. This observa- tion reinforces our hypothesis that the challenge presented by ImageNet-C may no longer be sufficiently difficult to rigorously test the robustness of modern models.
In contrast, models achieve much lower accuracy on LAION-C on average and exhibit more inter-model vari- ability. This showcases our dataset's ability to uncover nu- ances that remain hidden more saturated benchmarks. These performance differences are particularly obvious at higher intensity levels, illustrating LAION-C's potential to serve as a more challenging and insightful benchmark for evaluating robustness.
We also provide a detailed breakdown of non-averaged, dataset-specifie results in Fig. 11. We observe significant variability in the performance of different vision models across various datasets and distortion levels, highlighting the diversity in model robustness. These results further highlight the effectiveness of our datasets in eliciting dif- ferent responses from models of different architectures, pa- rameter scales, and training data sizes. This diversity is par- ticularly valuable for understanding which model designs are more robust to specific types of distortions, offering in- sights that are beneficial for advancing the state-of-the-arr model robustness.
3.2. Comparison to other OOD benchmarks
We conduct a direct comparison between LAION-C and other well-established out-of-distribution (OOD) datasets. As illustrated in Figure 4, LAION-C provides a more de- tailed resolution of model performance variances. This dataset captures broader variance in model performance, with a standard deviation of approximately ~27% com- pared to an average of ~10% in other common OOD datasets. Notably, LAION-C is evaluated on a 16-class ba- sis, which significantly fewer than the 200- 1000 classes used in typical OOD datasets, making the results even more remarkable.
In Tab. 10 in appendix, we present a comprehensive evalu- ation of our suite of models on LAION-C. We report each model's top-1 accuracy on the (undistorted) ImageNet val- idation set as baseline (Clean Accuracy) and the aver- age top-1 accuracy on LAION-C averaged across distor- tion types and intensity levels (LAION-C). We then break the latter down into the six distortion types. This enables a thorough comparison of model performance, highlighting which architectures generalize best.
100
80-
Accuracy (X]
60-
Intensity
20
3.4. Can LAION-C Be Solved?
IN
IN Skretch
LAIÓN
10.98% 10.21%
26.94%
Given the low performance of current state-of-the-art mod- els on LAION-C, one might wonder whether LAION-C is simply impossible to solve because the distortions de- stroy all information necessary for correct classification of the images. To disprove this hypothesis and highlight the validity of LAION-C as benchmark for evaluating model robustness, we conduct a fine-tuning experiment to assess whether the challenges posed by LAION-C are solv- able at all. Specifically,, we fine-tune a ViT-Huge model, which was originally pretrained with a CLIPPobjective on LAION-2B and then fine-tuned on ImageNet-225 and ImageNet- 1k. For this experiment, we use custom dataset sub-sampled from the ImageNet- training set and aug- mented with the distortions introduced in LAION-C. This dataset consists of over 336,000 images uniformly sampled across the 16 superclasses defined for LAION-C.
(standard destation of sccuracyl
Figure 4. LAION-C offers better resolution of model differ- ences. We tested 9 models pre-trained on LAION2B, evaluating them across all intensity levels if applicable. LAION-C C captures a broader variance in model performance, with standard devia- tion of ~27% compared to an average of ~00% in other common OOD datasets. Notably, LAION-C tested 16-class basis, while other datasets typically use 200-1000 classes, making this result even more remarkable.
3.3. Machine Performance
In Fig. 5, we compare model performance on ImageNet-C against performance on LAION-C. Evidently, the average model performance on ImageNet-C stays above or close to
6

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
100
Dataset
ImageNet-C
LAION-C
80
Model Class
Accuracy [%]
EVA
60
ConvNeXt
ViT
BEIT
40
DenseNet
WideNet
EfficientNet
20
ResNet
GPT-4o*
Gemini-1.5-Pro*
0
Average
1
2
5
Intensity Level
Figure LAION-C poses a greater challenge model robustness than ImageNet-C. We plot distortion intensity against each model's average accuracy. Visual foundation models evaluated on ImageNet-C maintain high accuracy, with minimal drop across increasing intensity levels. our LAION-C dataset, models exhibit a sharper decline accuracy, highlighting the benchmark's effectiveness in measuring model robustness.
3.5. Human and Machine Vision Discrepancy
Table LAION-C is challenging but can be solved by fine- tuning on the exact distortions. We compare the performance VITHL-P4--36CLLPPAANNN.
Accuracy Differences. In Fig. 6, we summarize how our suite of models performs in terms of classification curacy, compared to the human participants in psy- chophysical experiment. We report the best performances, since averages would be unfairly influenced by some older models we included as points of comparison. In Fig. 11, we provide a more detailed breakdown of performance by model. While human observers still outperform most vi- sion models images with Mosaic Glitched distortions, the best models match (or even slightly surpass) human per- formance. For distortions involving occlusion and lumi- nance manipulations, the vision models typically achieve higher accuracy than humans. Overall, current state-of- the- art vision models now match or even outperform human ob- servers in OOD scenarios, including on our synthetic dis- tortions, which they likely have never encountered during training- stark contrast to just few years ago, when humans were still vastly outperforming models (Geirhos et al., 2018; Dodge & Karam, 2019; Taori et al., 2020; Jang & Tong, 2024).
after fine- tuning it ImageNet-1k IK training images with LAION-C corrup- tions. As the performance after fine-tuning mun-tunig is high, this means that LAION-C, although challenging, remains solvable as it re- tains enough signal when applying distortions.
before
Accuncy 19)
Mosi
Natkel
Luninane
Geosacais
Sicker
IasgeNis-Vallte
453
614
346
Alter
906
03.6
958
91.8
90.
$14
992
As shown in Tab. 1, fine-tuning the model results in sub- stantial accuracy gains, which define upper bound LAION-C accuracy that no normal model can be expected to achieve. Notably, these accuracy gains are particularly pronounced on higher-intensity distortions, as detailed in Tab. 9, where accuracy is broken down by distortion in- tensity. The fine-tuned model likely achieves such good performance by employing un-human-liie (or "spurious") features, but the purpose of this experiment not to sug- gest that fine-tuning improve OOD robustness, but to quantify how much learn- able signal is left. LAION-C provides meaningful robust- ness tests without being intractable, making a valuable tool for more comprehensive evaluation of model perfor- mance under difficult conditions.
LAION is sensible approach to
Performance Complex Distortions. When analyz- ing more complex distortions such as Mosaic, Vertical Lines, and Glitched images (first row of Fig. 6), we ob- serve that human performance generally on par with the best performennn models. Especially at greater intensity levels, humans perform competitively, e.g., outperforming
7

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Mosaic
Glitched
Vertical Lines
100
80
Best Model
60
40
20
Accuracy (N]
Stickers
Geometric Shapes
Luminance Checkerboard
20
2
1
2
2
5
Intensity Level
Figure 6. Human machine accuracy all distortions. For each LAION-C distortion, we plot the distortion intensity against the accuracy of the best human and the best model this condition (for average human performance, see Fig. 11). The shaded regions indicate the 95% confidence intervals around the means. On the Mosaic, Glitehed and Vertical Lines distortions, humans and machines perform similarly, whereas the best model vastly outperforms the best human observer on the Stickers, Geometric Shapes, and Luminance Checkerboard distortions.
all models for the strongest Vertical Lines distortions. As we show in Fig. 11, the gap between humans and older models like the ResNet variants is particularly large on these complex distortions. However, modern model classes demonstrate substantial progress, approaching human-level performance even at higher intensity levels. While some margin for improvement remains, the narrowing gap sug- gests that achieving human-level robustness classifica- tion tasks is no longer the primary challenge for state-of- the-art architectures.
4. Discussion
Summary. Given that existing OOD benchmarks often no longer OOD for models trained web-scale datasets like LAION since distortions like blur and digi- tal corruptions are commonplace on the web, we here in- troduce LAION-C. LAION-C is benchmark designed to evaluate the robustness and generalization capabilities of modern vision models trained web-scale datasets. Our empirical results demonstrate that LAION-C is more chal- lenging for representative suite of vision models than the existing ImageNet-C benchmark, particularly at higher dis- tortion intensity levels. Our thorough human evaluation in a highly controlled psychophysical laboratory totaling 11,400 trials shows that the best models often outperform even the best human observers. While they do al- ways follow similar strategies (as indicated by error sistency analysis), this reassuring finding indicates that the best models have indeed substantially progressed in their ability to handle unexpected input and are not just getting better on in-distribution distortions. Given that the LAION- C benchmark dataset, by virtue of its construction, serves as a better proxy for model's 's ability to recognize objects despite unexpected distortion, recommend it as an OOD benchmark for current and future web-scale vision models.
Human-Machine Error Consistency on LAION-C. For more fine- -ra-ggaided analysis of the behavioral agree- ment between models and human observers, we calculate error consistency described in Sec. 2. As illustrated in Fig. 8 in appendix, there is high degree of variabil- ity in етrоr consistency between human observers and dif- ferent vision models ranging from to 0.4. This indicates that while model families such as VIT and EVA rival or sur- pass human performance, they are approaching the task uti- lizing different strategies than humans, demonstrating less human-likk like behaviors. The observed value range matches the one found in previous work for older models and differ- ent image data (Geirhos et al., 2021). This again suggests that while recent developments have boosted model perfor- mance, these models have not become more human-like like, as they follow alternative strategies.
While we have shown that LAION-C can effectively reveal shortcomings in model robustness, we have not yet fully explored why certain models underper-
Limitations.
8

---

LAION-C: An Out-of-Distribution Benchmark for Web-Seale Vision Models
form on specific distortions. Although our empirical results provide valuable insights, further investigation is required to clarify which visual cues the models rely under dif- ferent conditions. Such an analysis could inform the devel- opment of new inductive biases or architectural improve- ments, since a better understanding of these mechanisms could lead to improvements in both model interpretabil- ity and robustness. Given our current focus on introducing the dataset, this was not fully addressed here, but could be an area for future exploration. Furthermore, it is an open question what the performance limit on LAION-C looks like. Since fine-tuning models on LAION-C results in sig- nificant performance gains, particularly at higher distortion levels, there clearly is potential for optimization through advanced training techniques. However, how to further im- prove generalization across OOD scenarios, especially to enhance the models' ability to handle the novel distortions presented by LAION-C, remains an open question that war- rants further exploration. To retain its value as an OOD benchmark, LAION-C should not be used as a training or fine-tuning dataset (except for analysis purposes).
study. Detailed information was provided to each partici- pant beforehand, outlining the study's purpose, procedures and benefits, ensuring they were fully informed before agreeing to participate. Participants were also informed that they could abort the study at any time, without in- curring any negative consequences. Experimental data and contact information for the participants was stored in ac- cordance with GDPR. Participants were compensated with hourly base rate of 12 EUR and received bonus pay- ments based on classification performance, as is custom- ary in psychophysical experiments, so that the final reim- bursements exceeded the local minimum wage. This paper presents work whose goal is to advance the field of Ma- chine Learning. While we do not foresee any immediate negative societal consequences, the use of human partici- pants highlights the importance of maintaining high ethical standards. To enhance transparency and reproducibility, we also provided a detailed datasheet outlining the dataset's characteristics, collection methods, and intended use cases.
Code and dataset availability
The evaluation code for LAION-C is publicly available at: https://gi thub. com fean/FFafeiiiii IIAAOO-C. The LAION-C dataset is published on Zenodo. A link to the dataset is provided via the GitHub repository.
Conclusion and outlook. Just a few years ago, early in- vestigations into generalization abilities of deep neural ner- works showed humans vastly outperforming the best mod- els (Geirhos et al., 2018; Dodge & Karam, 2019). Fast- forwarding to today, LAION-C shows that the best models either match or outperform human performance on chal- lenging OOD distortions. This finding is reassuring in the light of growing concerns about the quality of existing eval- uation datasets, including the concern that OOD datasets like ImageNet-C may no longer serve their original pur- pose in the era of web-scale training datasets. Our find- ings indicate that the often super-human performance of modern models is achieved through super-human strate- gies: Models use variety of image cues- includdin,, in all likelihood, background pixels to perform well on some of our challenging datasets. Given their high performance across the board, they no longer rely on a single strategy that fails when faced with a test case that distorts a partic- ular image cue. This marks a paradigm shift in O0I) gen- eralization: From humans outperforming models to mod- els outperforming humans, from relying on a single strat- egy to a diverse set of multiple robust strategies, and from OOD benchmarking measuring progress towards human- like object recognition to better performance now indicat- ing super-human (in other words, less human-like) vision models.
Acknowledgements
We would like to thank Felix Wichmann for providing ac- cess to his psychophysies laboratory and valuable guid- ance, well as the students who took part in our study. We also thank multiple anonymous reviewers for their helpful feedback on the manuscript. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen Al Center, FKZ: O1IS 18039A WB ac- knowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1, via the Open Philanthropy Founda- tion funded by the Good Ventures Foundation and the Ger- man Research Foundation (DFG) through the SFB 1233, Robust Vision: Inference Principles and Neural Mecha- nisms, TP A2, project number: 276693517. WB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This research utilized compute resources at the Tübingen Ma- chine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Thomas Klein.
Impact Statement
We confirm that all experimental procedures involving hu- man subjects in our study had IRB approval. All partici- pants gave informed consent prior to their inclusion in the
9

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
References
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich- mann, F. A., and Brendel, W. Imagenet-trained enns biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Bao, H., Dong, L., Piao, S., and Wei, F. Beit: BERT pre- training of image transformers. In ICLR, 2022.
Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., and Oord, A. v. Are we done with imagenet? arXiv preprint, 2020. URL 2006. 07159.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W. Bethge, M., and Wichmann, F. A. Short- cut learning in deep neural networks. Nature Machine Intelligence, 2020a.
Biederman, I. and Ju, G. Surface versus edge-based deter- minants of visual recognition. Cognitive psychology, 20 (1):38 64, 1988.
Geirhos, R., Meding, K., and Wichmann, A. Beyond ac- curacy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In NeurIPS, 2020b.
Carandini, and Heeger, D. J. Normalization as a canon- ical neural computation. Nature reviews neuroscience, 13(1):51 12.51166,2226 2012.
Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., and Brendel, W. Partial success in closing the gap between human and machine vision. In NeurIPS, 2021.
Cohen, J. A coefficient of agreement for nominal scales, Educational and psychological measurement, 1960.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In CVPR, 2016.
Dodge, S. and Karam, L. Human and dnn classification performance on images with quality distortions: com- parative study. ACM Transactions Applied Perception (TAP), 2019.
Hendrycks, D. and Dietterich, T. G. Benchmarking neural network robustmess common corruptions and pertur- bations. In ICIR (Poster), 2019.
Doerig, A., Sommers, R. P., Seeliger, K., Richards, B., Ismael, J., Lindsay, G. W., Kording, K. P., Konkle, T., Van Gerven, M. A., Kriegeskorte, N., et al. The neuro- connectionist research programme. Nature Reviews Neu- roscience, 2023.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang. F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., al. The many faces of robustness: A critical analysis of nut-of-distribution generalization. In ICCV, 2021a.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner. T.,, Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Hendrycks, D., Zhao, K., Basart, S., Steinhardr, J., and Song, D. Natural adversarial examples. In CVPR, 2021b.
Hcusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NeurlPS, 2017.
Fang, Y., Wang, W. Xie, B., Sun, Q. Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Howard, A. G. Mobilenets: Efficient convolutional ral networks for mobile vision applications. arXiv preprint, 2017. URL https: wwwrxxiivorg/bbb.. 1704.04863
Fang, Y. Sun, Q., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 2024.
Huang. G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, 2017.
Fukushima, K. Cognitron: A self-organizing multilayered neural network. Biological cybernetics, 1975.
Jang, H. and Tong, F. Improved modeling of human vi- sion by incorporating robustness to blur in convolutional neural networks. Nature Communications, 15(1):1989, 2024.
Gebru, T., Morgenstern, J., Vecchione, B., Wort- Vaughan, J., Wallach, H., Daumé III, H., and Craw- ford, K. Datasheets for datasets. Communications of the ACM, 64(12):86-9,, 2021.
Kaufmann, M., Kang, D., Sun, Y., Basart, S., Yin, X., Mazeika, M., Arora, A., Dziedzic, A., Boenisch, E, Brown, T., Steinhardt, J., and Hendrycks, D. Test- ing robustness against unforeseen adversaries. arXiv
Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., and Wichmann, F. A. Generalisation in hu- mans and deep neural networks. NeurlPS, 2018.
10

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
preprint, 2019. URL https://arxiv.orr/aatt 1908. 08016
Peng, Z., Dong, L.,, Bao, H., Ye, Q. and Wei, F. Beit v2: Masked image modeling with vector-quantized vi- sual tokenizers, arXiv preprint, 2022. URL https:
Kellman, P. J. and Spelke, E. S. Perception of partly oc- cluded objects in infancy. Cognitive psychology, 15(4): 483-524, 1983.
/ //arxiv.org/ org/abs 20os/220 2208. 06366.
Radford, A., Kim, J. W., Hallacy, C.,, Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
Kleiner, M., Brainard, D., Polli, D., Ingling, A., Murray, R., and Broussard, C. What's new in psychtoolbox-3. Perception. 2007.
Recht, B., Roelofs, R., Schmidt, I.., and Shankar, V. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., and Lehtinen, J. The role of imagenet classes in fréchet in- ception distance. arXiv preprint, 2022. URL https: //arxiv. orq aaaabb/22205 6.026.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition chal- lenge. International journal of computer vision, 2015.
Liu, Z., Mao, H., Wu, C., Feichtenhofer. C., Darrell, T., and Xie, S. A convnet for the 2020s. In CVPR, 2022.
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajal- ingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott- Roy, J., Geiger, F,, et al. Brain-score: Which artificial neural network for object recognition most brain-like? BioRxiv, 2018
Mayilvahanan, P., Wiedemer, T., Rusak, E., Bethge, M., and Brendel, W. Does clip's generalization performance mainly stem from high train-test similarity? arXiv preprint, 2023. URL httpss:/aaxiv..oe 0310.09562.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image- text models. NeurIPS, 2022.
Mayilvahanan, P. Zimmermann, R. S., Wiedemer, T., Rusak, E., Juhos, A., Bethge, M., and Brendel, W. In search of forgotten domain generalization. 2024 Workshop on Foundation Models the Wild, 2024. URL https:/ http:::://ppeeevieewe net.forum? id id-Bc2p8T4V32.
ICML.
T, C. Glitch this, 2020. URL httpt:::s/tpttttieeee com/ Totall hww.aaoaeeeeaeeeeeeeeaaheaaaaaee
MIT License.
Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bring- mann, O., Ecker, A. S., Bethge, M., and Brendel, W. Benchmarking robustness in object detection: Au- tonomous driving when winter is coming, arXiv preprint arXiv:1907.0744: 077444 2019.
Tan, M. and Le, Q. V. Efficientnet: Rethinking model seal- ing for convolutional neural networks. In ICML, 2019.
Tanaka, J. W. and Presnell, L. M. Color diagnosticity in object recognition. Perception & Psychophysics, 61(6): 1 1140-1153, 1999.
Mintun, E., Kirillov, A., and Xie, S. On interaction between augmentations and corruptions in natural corruption ro- bustness. Advances in Neural Information Processing Systems, 34:3571-3588 2021.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. NeurlPS, 2020.
Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., and Kornblith, S. Human alignment of neural network representations. In ICLR, 2023.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.055: 055330 2024.
Nanay, B. The importance of amodal completion in every- day perception. i-Perception, 164):204 2018.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predic- tive power. Advances in neural information processing systems, 32, 2019.
OpenAl. Hello got-4o. https:/ / /oppaai...a.a../////// com/ index/hell pptt44/. 2024. Accessed: 2024-11- 19.
Wichmann, A. and Geirhos, R. Are deep neural networks adequate behavioral models of human visual perception? Anmual Review of Vision Science, 2023.
Parmar, G., Zhang. R., and Zhu, On aliased resizing and surprising subtleties in GAN evaluation. In CVPR, 2022.

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Wightman, R.
The timm (pytorch image models) leader-
board. https://hugg] ngg face.cc.ooooooo.oo cotccceessppcee/////c/ecs spaces/ Himm/leaderboaatt 2024.
Wightman, R., Touvron, H., and Jégou, H. Resnet strikes back: An improved training procedure in timm, arXiv preprint, 2021. URL https://///....... www.riiiwo..aaaboeeee 2110.00 00476.
Yamins, I.., Hong. H., Cadieu, C. F., Solomon, A., Seibert, D., and DiCarlo, J. Performance--pttmmized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 2014.
Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized out-of- distribution detection: A survey. International Journal of Computer Vision, (32((2))5635-
2024.
Zagoruyko, S. and Komodakis, Wide residual networks. arXiv preprint, 2016. URL https://axxiv.orr abs hbs/1605.071455
Zhai, X.,
Kolesnikov, A., Houlsby, N., and Beyer, L.. Scal- ing vision transformers. In CVPR, 2022.
12

---

LAION-C: An Out-of-Distribution Benchmark for Web-Seale Vision Models
A. Appendix
A.1. Related Work
OOD generalization ability of vision models. As deep learning has advanced to the point where models can reliably generalize to data that matches their training distribution or even exceed the quality of the original labels (Beyer et al., 2020), OOD-robustness, as measured by specifically designed OOD test sets, has moved to the center stage of computer vision research. In particular, ImageNer-C (Hendrycks & Dietterich, 2019), dataset containing images from the test-set of ImageNet to which various fairly natural corruptions such as blurring and pixelation were applied, has long been the gold standard for assessing DOD-performance to the point where data augmentations proposed to increase OOD robustness were found to only work well because they are similar to the ImageNet-C. corruptions (Mintun et al., 2021). In contrast, ImageNet-R (Hendrycks et al., 2021a) instead provides a more complex distribution shift by collecting different renditions of the target classes such as sculptures and paintings, instead of photos. In a similar vein, (Wang et al., 2019) introduce a dataset of black and white sketches matching the labels and scale of the ImageNet validation set, called ImageNet-Sketee A more subtle distribution shift which still caused considerable drops in model performance for ImageNet-trained models, was proposed by Recht et al. (2019). They collected ImageNetV2, new test set for ImageNet that should theoretically not differ from the ImageNet test set at all, because it was collected with very similar methodology, but revealed that models do perform slightly worse on ImageNetV2 than on the original test set. Hendrycks et al. (2021b) proposed two other OOD-test sets which do not rely on synthetic image manipulations but instead consist of natural images that are in some sense OOD relative to ImageNet, either by virtue of displaying object classes not present in ImageNet (ImageNet-O) or by showing an object of an ImageNet-class in a scene that is weird enough to fool most models (ImageNet-AA) What all of these datasets have in common is that, by design, they provide distribution shifts relative to ImageNet. But with the rapid evolution of deep learning, these traditional benchmarks have become less challenging for sate-of-thhear vision models trained on expansive web-scale datasets (Radford et al., 2021). While it is to some degree possible to predict model's s OOD generalization from its training set performance (Taori et al., 2020), the only reliable measurements of this capability stem from empirical evaluations of models on OOD datasets. Our work addresses this need by introducing LAION-C, a dataset that incorporates novel and complex synthetic distortions tailored to challenge even advanced vision systems.
Advancement in visual foundation models The rise of visual foundation models such as Vision Transformers (ViT) (Dosovitskiy et al., 2021), ConvNeXt (Liu et al., 2022) and BeiT (Bao et al., 2022) has redefined what constitutes standard performance across many visual tasks. These improvements in performance partially stem from architectural innovations and parameter optimization, but were mostly powered by the effective leveraging of unprecedented dataset sizes (Zhai et al., 2022). However, because visual foundation models were trained on web-scale datasets, the extent of their generalization capability remains underexplored.
Comparing human vs. machine perception. Deep Neural Networks were originally conceived as models of human vision (Fukushima, 1975) and were found to be the best available models for neuronal activity in the primate visual cortex (Yamins et al., 2014), even if not trained for this task. Today, there is a growing body of research dedicated to evaluating the adequacy of neural networks as behavioral models of human core object recognition (Doerig et al., 2023; Schrimpf et al., 2018; Wichmann & Geirhos, 2023; Muttenthaler et al., 2023). Building upon the findings of (Geirhos et al., 2021), who illustrate the narrowing of the behavioral gap between humans and machines in terms of their error consistency, our study further explores this dynamic utilizing LAION-C. We conducted a systematic analysis of differences in perception between human and machine observers, and assessed if the behavioral gap is closing further, as well as highlighting the persistent cognitive differences between humans and machines.
A.2. Experiment Procedure and Participant Incentives
Participant recruitment and setup. We recruited 20 participants (10 female) from the university student body via mailing lists. All participants were screened ensure normal or corrected vision and no color blindness, and gave informed consent participate. One participant was excluded post-hoc due to reporting extreme tiredness. Our experiments were conducted in darkened cabin, using 22" VIEWPixx 3D light LCD monitor (VPixx Technologies, Saint-Bruno, Canada) at a refresh rate of 120 Hz (scanning backlight mode on). The screen measures 484x 302 mm,, at a resolution of x920 1200 pixels. Stimuli were presented foveally in the center of the sereen, with a viewing distance of 65 em, resulting in 5 o of visual angle. In line with earlier experiments, the background was set to a grey value of 0.454 in the [0, 1] range. A chin rest was used to maintain a fixed viewing distance and angle. The experiment was implemented using the Psychophysics
13

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
DSS
2
C
Figure 7. Interface presented participants. This figure illustrates icon layout as displayed to participants during study. The grid is adapted from (Geirhos et al., 2018), while most of the categories and therefore symbols are different.
Toolbox (Kleiner al., 2007, version 3.0.12) in MATLAB (Release 2016a, The MathWorks, Inc., Natick, Massachusetts, United States) using 12-core desktop computer (AMD HD7970 graphics card "Tahiti" by AMD, Sunnyvale, California, United States) running Kubuntu 14.04 LTS.
The entire classification task, including both the warm-up and main experiment phases, was organized into 12 blocks. In each block, participants were shown images from the 16 superclasses for 2.5 seconds-a duration empirically determined balance efficient overall experiment length with sufficient exposure time allowing for viewing and consideration time. After each image, the 16 corresponding class icons were displayed screen, allowing participants 2 seconds classify each image into one of these categories. The icons were organized in layout that roughly grouped them by size and general category (e.g.. animals and objects), as illustrated in image Fig. 7. To encourage responses rather than leaving selections blank, message was displayed at the top of the screen 0.75 second before icon display time ended, prompting participants to make choice. At the end each block, a participant surpassed the 90% accuracy threshold calibrated using internal baseline performance data, they received an encouraging on-screen message ("Congratulations! You just earned some extra money!") along with a $0.50 bonus per block incentivize higher performance.
Warm-up session and main experiment. The experiment began with a 10-minute warm-up session, allowing partic- ipants familiarize themselves with the icon layouts and the classification task procedure through practice blocks. Each practice block contained 45 images, with block consisting of clean images and the other of augmented images. To avoid test-time adaptation, the augmentations used during the warm-up phase differed from those in the actual trials. The images used for the practice trials were also subsample of the ImageNet validation dataset, but did not overlap with those used in the main experiment.
Following the warm-up, main experiment proceeded consisting of blocks, each block comprising images. Each of blocks was augmented using consistent method, with balanced distribution across different intensity levels and superclasses. To avoid fatigue, participants were allowed unlimited amount of time rest between blocks and encouraged to rest their eyes accommodate elsewhere.
14

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
A.3. Error Consistency
Here, we provide a more detailed explanation of error consistency (EC), summarizing (Geirhos al., 2020b). The EC between two classifiers which both classified sequence of samples is obtained by first using the necessary ground-truth labels to assess which images cach observer classified correctly. A trial increases the agreement if both classifiers solved it correctly, or if they both failed (and decreases it if only one of them got the trial correct while the other one failed). One then calculates how much more agreement was observed between the two classifiers than observers relative to the agreement expected by chance, relative to the maximum possible delta. This done by calculating Cohen's ' Kappa (Cohen, 1960), which defined as (theoretical) expected agreement when modeling both observers as independent binomial observers. takes on values between -1 and 1, with higher x signifying higher levels of agreement, and k. of 0 indicating that a pair of observers does not agree more frequently than one would expect by chance, given their marginal correctness probabilities.
I-p. where Po is the (empirically measured) proportion of agreement-trials and pe is the
In this work, we calculate the error consistency between model responses and human classification data. To do this, we first collect all human responses. Since each human participant saw a fresh set of stimuli, we thus obtain exactly one human response per image. We then calculate each model's s EC to this list of human responses.
0.45
0.40
Error Consistency
0.35
0.30
0.25
0.20
VGG
ResNet
Inception
DenseNet
Wide ResNet
Efficientfommr
Gemini-1.5 5-Pro
EfficientNet
SWIN
Caformer
RegNet
DEIT
Convformer
EVA
GPT-40
BEIT
ConvNext
VIT
Model Type
Figure 8. Humans and models make different mistakes. We analyze the agreement of eтror patterns hetween different families of vision models (see Tab. 11 for a complete list) and human observers. The error consistency (k) could theoretically achieve maximum value of 1, but in line with carlier work (Geirhos et al., 2021), the EC values range between 0.15 and 0.45, indicating that behavioral differences between humans and machines are still quite large. Marker colors encode model families.
15

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
A.4. Augmentation Designs
A.4.1. MOSAIC
The image is divided into an n x n grid, where each tile is replaced by patch from the ImageNet validation whose average color best matches that of the tile. This patchwork creates mosaic effect that disrupts edges and textures while introducing contextually irrelevant information.
Level
2
3
4
5
4
6
16
28
Table 2. Tile sizes cach level.
A.4.2. GLITCHED
The original image undergoes an artistic digital corruption, with horizontal lines overlaying shifted image segments and color channel shifts. Here, region refers to randomly selected rectangular area of the image, and shift denotes the horizontal displacement (left right) of that region by a certain percentage of the image width. In addition, color channels are independently off'set by fixed number of pixels to further disrupt local spatial coherence. This dislocates the global contextual structure of the image. While the concept of such glitchy images has been explored in earlier work (Kaufmann et al., 2019), our transformation introduces a more intense corruption. Pixel shifts and color channel offsets are applied to random regions as follows:
Level
4
Shift
8% width
32% width
50% width
128% width
200% width
Regions
10
16
20
Offset
4 px
8 px
10 px
16 px
+20 px
Table 3. Glitch parameters each level.
The implementation is inspired by GitHub user "totallynotchase" (T, 2020).
A.4.3. VERTICAL LINES
The original image is transformed through a process of vertical deconstruction. It is first divided into multiple vertical sections, which are further subdivided along the y-axis into small segments called y-steps. In each of these segments, a short vertical line is drawn, where the direction of each line subtly reflects the local contour or edge orientation of the image. The color of each line segment is set to the average color of its corresponding region. This distortion retains the original colors but strips away local information, disrupting the contours and edges of the image and introducing visual discontinuity.
Level
1
2
3
4
5
Vertical Sections
224
178
112
84
60
Y-Step
1 px
2 px
4 px
6 px
px
Table 4. Vertical sectioning and step sizes at each level.
A. A.4.4 GEOMETRIC SHAPES
The original image is overlaid with overlapping geometric figures such as squares, circles, and stars. This visual clutter introduces local noise that obscures the main object, like the Kaleidoscope corruption from (Kaufmann et al., 2019). The number of shapes for each intensity level are shown as follows:
16

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Level
1
2
4
Shapes
150
300
600
800
1000
Table 5. Number of shapes at each level.
A.4.5. STICKERS
The original image is augmented by randomly placing 16 x 16 pixel image patches from the ImageNet validation set onto the image, following uniform distribution. This addition of visual elements masks features of the primary object and introduces deceptive new features.
Level
2
3
5
Patches
100
200
400
600
1200
Table 6. Number of patches at each level.
For an estimate of the occlusion ration of the objects per intensity level for stickers and geometric shapes distortions, see Tab. 8.
A.4.6. LUMINANCE CHECKERBOARD
The original image is divided into a 14 x 14 grid, with the luminance of each cell altered in checkerboard pattern. The stark luminance contrast between adjacent tiles and artificial grid boundaries makes this distortion challenging.
Level
3
Luminance Variation
50
150-100
=100-125
1125-550
- =150-255
Table 7. Luminance Variation at each level.
17

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Table 8. Occlusion ratio of objects in Stickers and Geometric Shapes distortions. We calculated the object occlusion ratio for the Stickers Geometric Shapes corruptions as an additional quantitative measurement distortion strength.
Intensity Level
Geometric Shapes (%)
Stickers (%)
61.88
65.83
2
72.51
76.52
3
85.35
86.19
4
90.16
89.54
5
93.21
91.63
Mosalc
Giitched
Vertical Lines
1.0
0.8
0.6
Pretrained VIT
(66
Rne-tuned VIT
Stickers
Geometric Shapes
Luminance Checkerboard
1.0
0.8
0.6
0.4
0.2
5
Intensity
Figure 9. LAION-C can be solved. For every distortion, we plot the accuracy of our reference model (VIT-H-P14-36--CIIPPLLA IN12K) before and after fine-tuning, in comparison to the best human participant for reference. Most distortions learned perfectly, only the Stickers and Mosaic distortions might have been too difficult at the highest intensity levels. Further performance gains might be possible with more careful fine-tuning.
18

---

LAION-C: An Dut-of-Distribbutin Benchmark for Web-Scale Vision Models
Table 9. Model (VTT) Accuracy Before and After Fine-TTuning on LAION-C. The high accuracies after fine-tuning indicate that even though the dataset is challenging. there is, in principle, enough signal left to perform well LAION-C.
Intensity Level
Accuracy Before (%)
Accuracy After (%)
89.0
96.3
2
71.9
93.0
Mosaic
3
35.8
88.7
14.3
69.6
14.7
47.7
1
79.9
95.9
2
70.1
94.9
Vertical Lines
3
50.8
94.1
4
36.1
92.4
5
19.4
90.0
95.9
98.6
2
86.2
97.5
Glitched
3
63.6
95.4
4
55.6
94.2
5
47.1
93.4
99.7
99.6
2
98.4
99.2
Luminance Checkerboard
3
95.1
98.8
90.7
98.5
5
56.6
92.5
30.9
99.4
11.2
98.6
Geometric Shapes
6.7
93.6
6.6
85.9
6.3
73.7
97.3
98.8
2
77.8
96.5
Sticker
3
28.7
63.7
4
14.9
31.8
5
8.1
14.3
19

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
A.5. Breakdown of model performance
To demonstrate the value of LAION-C benchmark for evaluating model robustness, analyze how model perfor- mance LAION-C correlates with that ImageNet-C. Grounding comparison in models that have demonstrated baseline level of robustness well-established benchmarks, apply threshold to include 40 models that achieved at least 60% accuracy on ImageNet-C.
As shown in Fig. 10, the majority of data points lie above the identity line representing performance alignment on LAION- and ImageNet-C. The gradual slope of the data points, combined with their positioning, indicates that models generally perform better on ImageNet-C, while their performance on LAION-C is more dispersed and often substantially lower. This broader distribution of performance highlights that LAION-C introduces more challenging distortions, prompting models to exhibit greater variability in robustness. The moderate Kendall's tau coefficient (r 0.66) between the models' performances LAION-C and ImageNet-C further underscores this, indicating notable pairwise differences in how mod- els rank across these two benchmarks, unearthing vulnerabilities that are less pronounced on ImageNet-C. These results demonstrate the necessity of LAION-C as complementary benchmark for more comprehensive evaluation of model robustness.
90
80
Accuracy on ImageNet-C ("Ik") [%]
70
DEiT
ConvNeXt
ViT
60
SWIN
Convformer
Caformer
50
RegNet
EVA
40
T 0.6555
30
30
40
50
60
70
80
90
Accuracy LAION-C [%]
Figure 10. Performance Divergence of Models on LAION-C and ImageNet-Ci (Ik classes). The figure illustrates the scattered perfor- mance of models across ImageNet-C and LAION-C, dataset, where Kendall's coefficient of and shallow slope indicate dispersed performance on LAION-C. To provide clearer trend and to better visualize the dispersion, supplement the suite of models with additional top- hop-peffrrmin models sourced from the timm leaderboard (Wightman, 2024), bringing the total number of models to 40 (see Tab. 11 for complete list).
Occlusion and Luminance Manipulations. For distortions involving occlusions, such as Stickers and Geometric Shapes, models usually match exceed human performance (see second of Fig. 11). One possible hypothesis is that this can be attributed to the robustness that models develop after e.g. masked image modeling (MIM) (Fang et al., 2023; 2024). The fact that models perform so much better than humans on partially occluded images implies that models use different features than humans. For example, for the Stickers distortion, certain ViT models outperform humans, likely due to their ability to focus on those parts of the image background that remain visible despite the occlusions. As shown in Fig. the stickers occlude nearly the entire image on higher intensity levels, and little no meaningful object information
20

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
is retained. Nevertheless, certain models are still able correctly classify the image based subtle background cues. This indicates that while models performing well, they may be doing by leveraging unintended shortcuts (Geirhos et al., 2020a), such as exploiting background information, when faced with severely occluded images. For the Luminance Checkerboard distortion, we observe that models from the ViT and EVA families outperforn humans by large margin. This advantage could potentially stem from their architectural features, such as self-attention mechanisms and patch-based processing (Fang et al., 2023; Dosovitskiy et al., 2021), which enable them to extract meaningful information from both light and dark regions independently, as well as handle subtle luminance variations. These capabilities give them clear edge over humans and older models.
Mosaic
Glitched
Vertical Lines
100
-.- Gemini-3
80
- GPT 1-4g"
+ DonsoNet
Efficien
ResNet
VIT
EVA
(N)
Hamans
Stickers
Geometric Shapes
Luminance Checkerboard
100
40
20
2
1
i
Intensity Level
Figure 11. Model performance on LAION-C. Analogous to Fig. 6, we relate distortion intensity level to classification accuracy for the different distortions, showing the different models individually. The shaded region around human performance corresponds to the 95% confidence interval, which omit for the models for better visibility.
21

---

LAION-C: An Dut-of-Distribution Benchmark for Web-Scale Vision Models
Table 10. LAION-C benchmark results. Numbers show top-1 accuracy in percent. ImageNer refers to model accuracy the (uncor- rupted) ImageNet validation set (values sourced from the timm leaderboard (Wightman, 2024)). For each coruupiion. we report mean top-1 accuracy across all intensity levels, with LAION-C the overall benchmark metric (averaged and Gemini 1.5 Pro were evaluated on 48,000 images, 100 for each class. For full names and descriptions, see Tab. 11 in the Appendix.
corruption types). GPT-4o
Model
ImageNet
LAION-C
Mosaic
Vertical
Glitched
Luminance
Geometric
Stickers
PWA.G-P14-5500-MM00NNI2XXX
89.8
48.8
53.6
70.8
97.2
81.0
53.4
90.1
66.8
53.6
58.2
78.2
93.6
76.4
40.6
V
88.6
57.3
45.2
51.2
69.8
88.2
64.4
24.6
WIL-PP4-23-CUPP-ppemmAANNZ2
88.3
57.8
52.6
49.8
68.2
98.6
55.4
22.4
VI-AFF2--8--CIILAAANNNNNI2
36.4
36.8
35.2
35.8
54.0
37.6
19.2
VTTBPP16-224-ugggg--NN111
85.5
47.1
46.4
42.8
62.0
71.4
42.4
17.6
BETT-22-L-P1--22-IIII
87.4
47.7
52.4
44.8
63.2
70.2
11.8
43.8
BFTT:x2-BB-16--224NNII
85.6
42.2
46.2
40.4
52.6
68.2
11.4
34.6
88.6
54.8
53.0
53.4
71.8
77.4
52.2
20.8
CunNEXX---CLIP-LIION-AAA IN12K
87.6
42.3
37.6
43.8
44.4
54.2
50.4
23.2
WRNIOI-2-TV-INIK
78.8
21.4
30.4
28.4
22.0
22.8
18.2
6.8
WRNSO-2-RACM-INIK
24.0
26.8
21.4
45.0
24.6
9.4
RN50-AI-INIK
81.2
19.9
28.0
18.8
20.8
23.4
21.2
7.0
EFF-B3-RA2-NE RRK-INNK
82.3
33.2
32.4
31.8
40.2
45.2
37.6
12.2
DN201-TV-INIK
77.3
19.2
28.6
26.2
13.2
23.2
16.8
7.2
DNI6I-TV-INIK
31.0
26.8
15.2
25.2
15.4
6.6
Go
54.1
42.8
45.4
65.1
80.1
$4.2
36.5
Gemini 1.5
50.2
34.9
37.0
46.2
84.4
60.9
38.1
Best human observer
58.0
55.3
78.7
63.4
40.4
35.7
Average human observer
47.0
50.8
43.6
71.0
53.1
34.3
29.1
22

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
A.6. Evaluating VLMs
To evaluate GPT-4o (OpenAI, 2024) and Gemini 1.5 Pro (Team et al., 2024) on LAION-C, we decided test a random subsample of the full dataset, consisting of 100 images per category, which were then tested all corruptions and intensity levels, resulting in total of 48,000 images. For ImageNet-C, we limited ourselves to only 10 images per class, get an initial ballpark estimate of performance.
We employed the following system prompt, in line with our human experiments, during which participants were also shown examples:
System Prompt:
You are an image-recognition API.
You are always asked to classify the main object of images into one of 16 mutually exclusive categories.
In some images, distortion may be so strong that you might not recognize anything. you're unsure, provide your best guess you always have to pick exactly one the 16 categories. primate, dog, cat, bird, fish, snake, butterfly, fruit, boat, vehicle, chair, ball,
The 16 categories bottle, instrument, timekeeper, tool.
Here is list of characterizations of every such category:
primate: a primate, like e.g. monkeys, chimpanzees, Orang-Utans Utans etc. dog: a dog, like e.g. german shepherd, labrador, golden retriever etc.
cat: a cat, like e.g. domestic cat, lion, cheetah etc.
bird: a bird, like e-g. songhird, eagle, chicken etc.
fish: fish, like e.g. trout, shark, whale etc.
snake: snake, like e.g. viper, cobra, seasnake etc.
butterfly: a butterfly, like e.g. monarch, cabbage butterfly, ringlet etc.
fruit: fruit, like c.g. apple, orange, pincapple etc.
boat: a boat, like e.g. ship, gondola, fireboat etc.
vehicle: a vehicle, like e.g. truck, van, sports car etc.
chair: chair, like e.g. bench, throne, couch etc.
ball: a ball (or person playing with a ball), like e.g. soccer ball, football, tennis ball etc.
bottle: bottle, like e.g. water bottle, jug, pill bottle etc.
instrument: musical instrument (or person playing an instrument), like e.g. sax, flute, harp etc. timekeeper: timekeeper, like e.g. clock, watch, sundial etc.
tool: a tool (or a person using a tool), like e.g. hammer, power drill, chainsaw etc.
Since you are an API, you always respond with minimal messages that contain exactly one word, which is the category name.
User Prompt:
What is the main object in this image? Categories are: primate, dog, cat, bird, fish snake, butterfly, fruit, boat, vehicle, chair, ball, bottle, instrument, timekeeper, tool.
23

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
The image is nighly pixelated, but can discern the general shape dog, especially the head and front legs. The color pattern suggests dog's
True label: fish Predicted label: dog (1)
The Image seems to show fruit, possibly an apple, based on the red and green colors present in the picture.
True label: chair Predicted label: fruit mosaic (1)
The distorted image appears to show the outline of bird with wings outstretched. The pointed shapes resemble wings, and the overall posture suggests bird in flight.
True label: ball
Predicted label: vertical lines (2)
can pliers in the center of the image. Pliers
True label: tools Predicted label: tools (2)
image appears show potentially chicken, due to the presence of orange feathers and what looks like wattle. The distortion makes it difficult be certain, but the color and shape suggest
True label: dog Predicted label: bird giltched (1)
any other category.
round shape presence appears drumstick suggest that this image represents a drum, which is musical Instrument.
Predicted label: instruments glitched (1)
Although the image heavily distorted, it appears to show the face of primate. The discemible features, the eyes ridge, suggest primate's facial structure.
True label: cat Predicted label: primate glitched (2)
Figure 12, Visual Reasoning in Gemini. We provide examples of visual reasoning Gemini-1.5-Prrs consisting LAION-C sample, the reasons for classification that Gemini provided and meta-information (like the final label, the ground-truth label and corruption details). In line with our findings about Error Consistency (see Fig. 8), the model is frequently confused by visual impressions that are trivial for humans to decode. Instead, it hallucinates unreasonable image properties.
24

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
The image shows a shape that resembles the outline a bottle, with a neck and body typical of bottles, despite the heavy distortion and colorful overlay.
True label: Predicted label: bottle geometric shapes (3)
The metallic object with handle, resembling a hammer, which is common tool. The shape and context suggest it used for hitting or driving nails.
label: chair Predicted label: tools glitched (1)
The image appears to be mosaic or collage that forms the shape of primate, likely monkey overall structure body of a primate, which is why this category is chosen.
True label: fish Predicted label: primate mosaic (2)
features
The image appears to depict stylized or pixelated representation primate, likely monkey ape, and overall
True label: snakes
Predicted label: mosaic (2)
shape.
The image shows a shape and texture that resembles fish, with streamiined body and scales visible, The colors patterns typical fish, making the most likely category.
True label: chair
Predicted label: fish vertical_iines
The image appears have distorted outline that resembles the shape of primate, such as monkey The contours suggest facial features and posture typical primates.
True label: battle Predicted label: primate vertical lines
The Image contains muitiple small Images, many which depict butterflies. The presence wings and colorful patterns typical of butterflies is noticeable, making "butterlly' the most fitting category.
True label: Predicted label: butterfly mosalc
Figure 13. Visual Reasoning in GPT. Figure analogous Fig. 12. Like Gemini, GPT has tendency to hallucinate visual impressions. Notably, when the model makes mistakes the sticker-corruttion, it typically led astray by the contents of the sticker-images, ignoring the background completely and failing to perceive the global structure of the image, behavior that Gemini exhibits well.
25

---

LAION-C: An Out-of-Distribution Benchmark for Web-Seale Vision Models
A.7. Models
Table 11. Model overview. For each model used in our evaluation, we show the full model names, as used in timm, an abbreviated name used in the main text and description of the model. While the first 16 models were used in all analyses and figures, the rest was only used for selective analyses such as Fig. 8,
Abbreviation
Full Model Name
Deseription
DWA P14 560 MXM INIIK
glant puich14 5601m30m in22k inlk
EVA model, patch size 14, pre trined with masked model ing (MIM) on Merged 30M dataset, ImageNes 1s (Fang et al., 2025).
ImageNett 22k and
eva02 large pucch a 448.mmm mim m38m EVA12 L P14 448 MIM MISM IN22K
in22k inik
EVA22 lurge model patch size 14, pre trined with masked image mod (MIM) Merged 3MM dataset, fine tnne on Imugeleet 22K and ImageNect-Bl (Pung al., 21144)
VIT LE P14 336 CLIP LAION IN12K
vit huge patch14 4.cip 236.laion2b it in12k inik
Vision Iransormer (VIT) huge model. pacc size 14, pre traindd on LATON 2B dutaset using OpenCLIP, line timed on ImageNet 12k and ImsogoNet-1 (Doswinkkiy et al. 2021).
224 CLIP OPENAI
vit.Jarge.pooch 4uclip waddiip224..pee 2t....2nii.....oooo iik.iikk
Vision Transfommer large model, patch 14, pre trined on 400M using CLIP, fine tuned on ImngeNet- 12k and ImageNee Ik (Doooviskiy et al. 2021).
sit Juuhscpptcch22.ol _2llpp.pp.....__ 184 laionth i
Vision Tranformer base model, patch sine 32, protoained on LAION- using OpenCLJR.ine-tmmon (Dowwitskly et al. 2021).
ImageNeet- 12k ImugeNer
VTT---P1--22-AAXEEE--
sit patchhh 6 224 angrog (21.k Juiin
Vision TrannTTann Tranffrmee hase model, patch size 16, trained on ImageNes-2111 and fine tined ImageNet- Ik (Dosovitskly al. 2021). lange
DE-TV2---P1--22--N-
MaieiJirggggeed gutchi 66_234 Ii
trsined ImugoNes- fine-tned ImageNas-22 and ImsggNNs- It (Roo sl., 2022, Pesg etal. 2022) BETVV base model, putch size 16, tmined on IaugeNes- lk, fin--tued on EmtggeNe-22X and ImageNet- it (Roo et sl. 2022; Peng stal, 2027)
BEITV2-B-P PI1-----III
beiv2_ hase 1a 234inkk siilkk
CONYAXXI -CL -IOLAII-I
CanuNeXi sslagg model, pre-trained using OpenCI LA00N288. fine-tuaed ImugeNes--1 (Liu al., 2023;
CONV B. CLIP LAIIN--UUGEES IN12K
convaest base luionZb angreg it inl2k inlk 384
ConoNeXi base modet,pre trinedd using OpenCLIP on LAJON-20 28. line-tuned ImugeNict-12 12 and ImageNes- I1 (Lis a al. 2023).
WRNIUI-2-TV-III
wide rew.ett0111111. 2at.tooo1...vvveee jink
ResNes- tot model, mained ImageNet- with original turcavision model weight (He al., 2016: Zagoruyko Komodakis. 2016).
WRNSD-22 2 RACMM-NIA
wide rww.ecto. 2.acmo inlk
Wide ReaNer 50 model, tmined with KandAugment RACM eecipe ImageNet (He al., 2016: Zagonyko Komodakks, 2016). ResNet model trained on ImageNet (Ile et al., 2016; Wightman et al., 2021).
RN50 INIK
resnet50.al link
DFF B3 RA2 INIK
etticientnet bd.r2 inlk
EfficientNet B3 model, trained with RandAugment RA2 recips on ImageNet (Tan 2019).
DNIOI-TY-INIK
densemer201 iv Ik
DensoNet-200 201, DenoeNet pre-tninee ImageNee- (ffunng 2017).
DNI6I-TV TIIK
densemetóól.vv atv inik
DenseNet 161, DemoNNet model pre trained on ImageeNet Ik (llunng et al., 2017).
apt 3p1-4--2024055-66
writing, moot recent stapshox OpenAl's flushhp modill (OpenAI, 2024). Only evalusted on 48,000 LADN-C samples and 12,000 ImageNes-C samples.
Genini--155PP
gemini- 1-5-poo-
At the time of willing, the moot recent suble version of Google's Gem- model (Tom al., 2024), Only asalusted 48 LAION-C sumples and 12,000 ImageNas-CC samples.
conveestv2 pico Semae dt inik
convoestv2..k 2...eent....nyyy Temse inlk
base.tb db in23k it inlk
convnest. large mlp. laion augreg diinkk_388 384
Jorvnent.Japee..o corrpr_mpp.ii cwp.hiiiiiiionnnbb swmpssspuuiii ftni 25..in 384
convoext. wiw.ee...nyyln12itt...ll.. Innk
small Db in22k in1k
convnet_sslage .ilk
convnest_smallle int _1______n11__184
convnesrv2 2.lre large Semae it in.nn22 inlk
vit MiLhetwiss.ppe ppachhhhh wwt..ppp..26666..m 256.
sit_modiiumd_ wf.ppptchhh......eor 2ptp_226666....0
vit setses fulggpttiilileelppppp 2kt,
vit hitmmediumd.ppi 16 kurgg-gpppppp wwp.256.dhb2......
vit situmddiundd patch 66 furog 256.pp6pp666600000 aib ini sit tippoeeegse wes.___cchl__eggg...... 25
vit betwist. patch 16 rope rope.mg aop 256 stb inlk
vit BiLhewist parchi cnpy 256 whi jni ninkk
maxxvity2 mip. haseerww 384 ow Jin 2k jaakk
19919 batt Jolk
Je.jjik
sagnett_11800ss 112800swggg... jiinkkk
regnety awul inik
incoption v3.st ade inlk
bese pollDeeepstcchhl sescch66_224innne wtw.kinkkikkaaaak
beit large putch1..112aa222 512.a22k
It la22i, inlk
deli3 large paich16 354.tb in23k inlk
base Sestilleddpaccclle 221.fh.h..24..iikkkkkk
swin base putch4 window7 224 in2Ik inik
swim2 base window12hu24 192mm in22k It inik 2.m.2.2ang www.2...aee.iinn window 22c24__ 10334_11900 1926634 ma..060.mm.iii2.. fiikk lurge
336.n22k inlk
conviormrr b36.sall in22k
cnformer h36 2... 2ts.alin222....iik..... fiinkk388
offisieffrrmee222..000 minkiiiiin
26

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
A.8. Datasheet for LAION-C
We here include Datasheet for LAION-C following the template proposed by (Gebru et al., 2021).
Motivation
Does the dataset contain all possible instances or is it a sample (not necessarily random) of in- stances from a larger set? sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic cov- erage)? ? If so, please describe how this representa- tiveness was validated verified. If it is not represen- tative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
the dataset is a
For what purpose was the dataset created? Was there a specific task mind? Was there a specific gap that needed to be filled? Please provide a de- scription.
The LAION-C dataset was created to serve bench- mark for evaluating the robustness and (OOD) generalization of large-scale vision models. It can also be used to study the difference between human and model perception.
The dataset is a sample of the ImageNet validation set and only contains 4.368 of the 50,000 images. As such, LAION-C is not representative of ImageNet, because it only consists of coarse superclasses. This decision was made to facilitate measuring human classification perfor- mance on LAION-C, which would not be possible with the fine-grained classes of ImageNet.
Who created this dataset (e.g., which team, re- search group) and on behalf which entity (e.g., company, institution, organization)?
The dataset is created by authors of this paper.
Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or fea- tures? In either case, please provide description.
Not applicable.
Each instance consists of RGB image, as well as meta- data about the ground-truth class, corruption type, and severity level, which are simply part of the filename.
Any other comments?
None.
Is there a label or target associated with each in- stance? If so, please provide a description.
Each image is labeled with its superclass (one of 16) and can be traced back to original ImageNet class label.
Composition
What do the instances that comprise the dataset represent (e-g., documents, photos, people, coun- tries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
Is any information missing from individual in- stances? If so, please provide a description, explain- ing why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
The instances in the LAION-C dataset represent images grouped into 16 superclasses with various synthetic distor- tions applied them at 5 severity levels. Each superclass contains 273 images, and the distortions include mosaic effects. glitched images, vertical lines, geometric shapes, stickers, and luminance checkerboard patterns.
No information is missing from individual instances as each image in the dataset is synthetically altered and la- beled with the type distortion and its severity, ensuring comprehensive data for evaluation purposes.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these re- lationships are made explicit.
How many instances are there in total (of each type, if appropriate)?
total, LAION-C consists of 131,040 images. (16 classes 273 images 6 corruptions x severity levels.)
The dataset does not contain explicit relationships between individual instances such as social links or ratings since it
27

---

LAION-C: An Out-of-Distribution Benchmark for Web-Seale Vision Models
primarily focuses on image recognition and distortion type evaluation without any relational context between the im- ages.
ImageNet are manually filtered to exclude any content that could be considered disturbing.
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
Are there recommended data splits (e.g., training, development / validation, testing)? If so, please provide description of these splits, explaining the rationale behind them.
Yes, the LAION-C dataset relates to people to some ex- tent as it includes images from ImageNet, some of which feature human faces and figures. While the primary focus of the dataset is not on the individuals depicted or on ana- lyzing human-specinee data, the presence of human images means that the dataset does relate to people indirectly.
Since the dataset is primarily used for benchmarking pur- poses, splitring specifics are not provided. Essentially, the entire dataset is a validation set.
Are there any errors, sources of noise, or redun- dancies in the dataset? If so, please provide a de- scription.
Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a de- scription of their respective distributions within the dataset.
The dataset is designed to introduce controlled noise through synthetic distortions test model robustness. There are no unintentional errors redundancies; all mod- ifications serve the purpose of benchmark evaluation.
The LAION-C dataset itself does not explicitly identify subpopulations by age, gender, or other demographic char- acteristics as part of its core design. However, since it in- cludes images from ImageNet, which may contain human faces, there is an implicit presence of such demographic data.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e-g., web- sites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external re- sources and any restrictions associated with them, as well as links or other access points, as appropriate.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
While the primary intention of the LAION-C dataset not to facilitate the identification of individuals, it incorporates images from ImageNet, which may include human faces.
The dataset is entirely self-contained.
Does the dataset contain data that might be con- sidered sensitive in any way (e.g., data that re- veals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union mem- berships, locations; financial or health data; biometric genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a descrip- tion.
Does the dataset contain data that might be con-
sidered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals non- public communications)? so, please provide a description.
The dataset does not contain confidential data it is based on publicly available ImageNet data.
While the LAION-C dataset primarily features synthetic distortions applied to images for technical analysis, it in- cludes images sourced from ImageNet that may contain human faces. These images can indirectly reveal racial or ethnic origins due to the diversity of individuals depicted. However, there is no explicit focus on collecting ana- lyzing data related to sexual orientations, religious beliefs, political opinions, union memberships, specific locations, financial or health data, biometric or genetic data, govern- ment identification numbers, or criminal history. The in-
Does the dataset contain data that, if viewed di- rectly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
The dataset does not contain offensive or disturbing con- tent as it focuses on visual distortions applied to non- sensitive images. Additionally, the images sourced from
28

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
clusion of human images is incidental and not intended for any analysis related to these sensitive aspects.
how were they compensated (e.g., how much were crowdworkers paid)?
Details are managed by the original collector for Ima- geNet.
Any other comments?
None.
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., re- cent crawl of old news articles)? If not, please de- scribe the timeframe in which the data associated with the instances was created.
Collection Process
How was the data associated with each instance acquired? Was the data directly observable (e.g.,
The source dataset for the creation of LAION-C was the 2012 HLSVRC validation set ("ImageNet") which was collected over several years. The distortions applied in LAION-C were created specifically for benchmarking pur- poses the time of dataset development (2023 2024), which do not coincide directly with the original image col- lection periods.
raw text, movie ratings), reported by subjects (e.g.,
survey responses), or indirectly inferred derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported derived from other data, was the data validated / verified? If so, please describe how.
by subjects or indirectly inferred /
The data for each instance in the LAION-C dataset is de- rived from ImageNet, where images are directly observable and not reported by subjects or inferred.
Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review pro- cesses, including the outcomes, as well as link or other access point to any supporting documentation.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sen- sor, manual human curation, software program, software API)? How were these mechanisms or pro- cedures validated?
The original ImageNet dataset underwent various ethical and review processes during its development, details managed by the original collector for ImageNet.
First, 16 sensible high-level classes were selected that the authors deemed suitable for humans to recognize in psy- chophysical experiments. These classes are: ball, bird, boat, bottle, butterfly, car & truck, cat, chair, dog, fish, fruit, instrument, primate, snake, timekeeping, and tool. Then, 200 classes from the original ImageNet-Ik set were se- lected that can constitute these high-level classes. From the pools of validation set images, 500 images were randomly selected per superclass. These images were then manually filtered to include only images that fall clearly into one of the 16 superclasses (i.e. an image showing both a ball and dog would have been filtered out to ensure clean class labels).
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
Only indirectly. LAION-C includes images from Ima- geNet that feature human faces and figures.
Did you collect the data from the individuals in question directly, obtain it via third parties or other sources (e.g., websites)?
Not applicable.
Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the no- tification itself.
If the dataset is sample from larger set, what was the sampling strategy (e.g., determinis- tic, probabilistic with specific sampling probabili- ties)?
Not applicable.
See previous question. Candidate images from the con- stituent classes were sampled randomly with uniform prob- ability.
Did the individuals in question consent to the col- lection and use of their data? If so, please de- scribe (or show with screenshots or other information) how consent was requested and provided, and pro- vide a link or other access point to, or otherwise re-
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and
29

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
produce, the exact language to which the individuals consented.
can be found here: https: /tt:://wwwww -eetaaaeeeeeeeeeeeeeee org/
down php
Not applicable.
Is the software used to preprocess clean label the instances available? If so, please provide a link or other access point.
If consent was obtained, were the consenting in- dividuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point the mechanism (if appropriate).
Yes, the preprocessing, cleaning, and labeling of the dataset instances were conducted using Python. The code used for these processes accessible via https:// github.
Not applicable.
Any other comments?
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., data protection impact analysis) been conducted? If so, please provide description of this analysis, in- cluding the outcomes, as well as link or other access point to any supporting documentation.
None.
Uses
No specific data protection impact analysis has been con- ducted for the LAION-C dataset as its primary modifica- tions involve applying synthetic distortions like glitches to the images for technical benchmarking purposes. These al- terations do not fundamentally change the nature of the data regarding privacy or ethical concerns beyond their original use in ImageNet.
Has the dataset been used for any tasks already? If so, please provide a description.
Yes, the LAION-C dataset has been utilized evaluate the robustness and out-of-distributtion (OOD) generaliza- tion capabilities of large-scale vision models.
Is there a repository that links to any or all papers or systems that use the dataset? If so, please pro- vide a link or other access point.
Any other comments?
None.
https: /
What (other) tasks could the dataset be used for?
Preprocessing / cleaning / labeling
Beyond benchmarking vision model robustness, LAION- C could be used in studies investigating the effects of image distortions human perception.
Was any preprocessing cleaning labeling of the data done (e.g., discretization or bucketing, to- kenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. not, you may skip the remainder of the questions in this section.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed / cleaned labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could re- sult in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other un- desirable harms (e.g., financial harms, legal risks) If so, please provide description. Is there anything future user could do to mitigate these undesirable harms?
Images were resized to 256x256 pixels and center-cropped 224x224 pixels, as is common for ImageNet. Images were filtered manually to ensure clean labels described above.
Was the "raw" data saved in addition to the pre- processed cleaned / labeled data (e.g., to sup- port unanticipated future uses)? If so, please pro- vide a link or other access point to the "raw" data.
Given that the base images in the LAION-C dataset are sourced from ImageNet, which already publicly avail- able, the additional risk for harm is negligible.
No, LAION-C only consists of the modified images, but every filename can be uniquely traced back to the parent image from the ImageNet validation set, which
Are there tasks for which the dataset should not be used? If so, please provide description.
30

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
We would recommend using the LAION-C dataset for fine-tuning machine learning models, due dataset size.
and provide a link other access point to, other- wise reproduce, any supporting documentation.
Since the images are modified ImageNet images, the re- strictions of the ImageNet license apply.
Any other comments?
None.
Any other comments?
Distribution
None
Will the dataset be distributed to third parties out-
Maintenance
side of the entity (e.g., company, institution, orga- nization) on behalf of which the dataset was cre- ated? If so, please provide description.
Who will be supporting hosting maintaining the dataset?
The LAION-C dataset will be made publicly available, al- lowing for distribution third parties outside of the origi- nating entity.
The dataset maintained by the authors of the associated paper.
How can the owner curator / manager of the dataset be contacted (e.g., email address)?
How will the dataset will be distributed (e.g., tar- ball on website, API, GitHub) Does the dataset have digital object identifier (DOI)?
For questions regarding the dataset, please contact Fanfei Li at <first. last etuubingen.nnnn. www.genn...........m. mpp .d>>.
The dataset is accessible via Zenodo: https:// zenodo. org/ httg///ecorrs/14051110
Is there an erratum? If so, please provide link or other access point.
When will the dataset be distributed?
There is not explicit erratum for now.
The dataset is already distributed.
Will the dataset be updated (e.g., to correct label- ing errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and or under applicable terms use (ToU)? so, please describe this license and ToU, and provide link or other access point to, otherwise reproduce, any relevant licensing terms ToU, well as any fees associated with these restrictions.
Information will be provided upon publication.
If the dataset relates to people, are there appli- cable limits on the retention of the data associ- ated with the instances (e.g., were individuals in question told that their data would be retained for fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
LAION-C will be available under a CC BY-NC 4.0 license, allowing non-commercial use with proper attribution only, to ensure compliance with the original ImageNet license.
Have any third parties imposed IP-based or other restrictions on the data associated with the in- stances? If so, please describe these restrictions, and provide link or other access point to, or other- wise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
Not applicable (beyond agreements made for ImageNet).
Will older versions of the dataset continue to be supported / hosted maintained? so, please de- scribe how. If not, please describe how its obsoles- cence will be communicated to users.
The original ImageNet data is subject to terms of access that limit its use to non-commercial research and educa- tional purposes only. The full terms of access can be found here: https:/ /tts::/wwwr /www. image-neee net. org/ prg/download. php
Should newer versions of the dataset be created, older ver- sions will continue be available via Zenodo.
Do any export controls or other regulatory re- strictions apply to the dataset or individual in- stances? If so, please describe these restrictions,
If others want to extend / augment build on con- tribute to the dataset, is there a mechanism for
31

---

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
them to do so? If so, please provide description. Will these contributions be validated / verified? If so, please describe how. If not, why not? Is there pro- cess for communicating / distributing these contribu- tions to other users? so, please provide a descrip-
for example by contributing their own corruptions. While there is automatic mechanism (such as publicly accessi- ble version control, e.g. via Github) for this, we encourage interested parties to reach out to the authors.
tion.
Any other comments?
We encourage other researchers to build on LAION-C.
None.
32


Reviews:
[{"review_id": "5SEfmJltez", "reviewer": "Reviewer_qKNV", "summary": "This paper introduces LAION-C, a new benchmark dataset for evaluating out-of-distribution (OOD) robustness of web-scale vision models. The authors argue that existing benchmarks like ImageNet-C are no longer sufficiently OOD for models trained on massive web datasets like LAION, as these models are likely exposed to similar corruptions during training. LAION-C proposes six novel, synthetic distortion types designed to be genuinely OOD even for web-scale models. The paper presents evaluations of various state-of-the-art models, including large language models with vision capabilities, on LAION-C and compares model performance to human psychophysical data, suggesting a paradigm shift where models are now matching or exceeding human OOD robustness in certain scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nthank you for your insightful comments! We are delighted that you find our work to have a **good experimental design** and consider it **relevant to the community**. \n\n\n*Q: “More ImageNet OOD variants could have been studied / compared with”*\n\nA: Thanks for this excellent suggestion. We have added a comparison of LAION-C and several well-established OOD datasets to the paper, see https://ibb.co/HfZmLkTd, which shows that LAION-C captures the variance in model performance better than other datasets, with a standard deviation of ~27%, whereas other common OOD datasets, on average, have only ~10%. LAION-C is tested on a 16-class basis, while other datasets typically use 200-1000 classes, making this result even more remarkable. We want to note, however, that the point of LAION-C is precisely not to capture “naturally occurring” perturbations (see next question). We also added the missing reference to ImageNet-Sketch you pointed out (note the related work section in Appendix A.1, where we already refer to ImageNet-A).\n\n\n*Q: “[T]he synthetic nature of the perturbations questions whether they represent real world occurrences [...]”*\n\nA: Indeed, LAION-C is designed to be synthetic, just like ImageNet-C for ImageNet. “Natural” images as used by other common OOD datasets are scraped from the web, making them no longer OOD in an era where large multimodal foundation models are trained on nearly the entire web, for multiple epochs: as of 2025, we need to assume that most natural images found on the web were part of the training corpus. Robust representations, however, should generalize to challenging images (e.g. identity-preserving transformations) no matter the type, both synthetic and natural—just like human perception is robust to many different types of OOD data. Being able to handle challenging synthetic input typically transfers to unexpected natural input, as evidenced by the “sim2real” line of work e.g. in robotics and autonomous driving.\n\n\n*Q: “Have the authors considered constructing a OOD test set in a similar manner to the ImageNet-A dataset but for LAION to test real world failure cases?”*\n\nA: Thank you for the suggestion. In the current era of large-scale training, we have to assume that models will eventually be trained on the entire internet, hence scraping the web for benchmark images, as was done for ImageNet-A, is no longer a feasible choice for obtaining an OOD dataset.\n\n\n*Q: “The paper oversells the paradigm shift claim as the perturbations proposed are in the same vein as the ImageNet-C corruptions.”*\n\nA: This might be a misunderstanding. The “paradigm shift” we refer to is the shift from humans outperforming models (as observed in prior work [1,2,3,4,5]) whereas now, based on our carefully collected human comparison data for LAION-C, the best models outperform humans. Apologies if that wasn’t clear, we’ll make sure to highlight and explain this better.\n\n[1] Partial success in closing the gap between human and machine vision (Geirhos et al., 2021) \n\n[2] Comparing deep neural networks against humans: object recognition when the signal gets weaker (Geirhos et al., 2017)\n\n[3] ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints (Dong et al., 2023)\n\n[4] Recognition in Terra Incognita (Beery et al., 2018)\n\n[5] Why do deep convolutional networks generalize so poorly to small image transformations? (Azulay & Weiss, 2019)", "claims_and_evidence": "The authors report that existing OOD evaluation benchmarks such as ImageNet-C are not truly OOD anymore as LAION-based datasets basically train on whole web. This fact is anecdotally supported in Figure 1, performance of zero-shot models is compared between the ImageNet-C test set and the LAION-C one to show LAION-C is harder to solve and finally, a FID score is computed to support representation differences between datasets. \n\nThese claims adequately support the limitations of ImageNet-C and the added difficulty of LAION-C for zero-shot models.", "methods_and_evaluation_criteria": "The authors evaluate a large suite of vision models as well as some experiments with human discriminators to put in parallel the difficulty increase between human and machine when increasing the perturbation intensity.\n\nMore ImageNet OOD variants could have been studied such as  ImageNet-Adversarial [1] or ImageNet-Sketch [2] which would represent more naturally occurring OOD perturbations.\n\n[2] Hendrycks, Dan, et al. \"Natural adversarial examples.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[3] Wang, Haohan, et al. \"Learning robust global representations by penalizing local predictive power.\" Advances in neural information processing systems 2019.", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "Good design, other benchmarks OOD than ImageNet-C should be added and compared with.", "relation_to_broader_scientific_literature": "This research is relevant to the community as it shows the limits of the ImageNet-C benchmark for OOD robustness and proposes an alternative.", "essential_references_not_discussed": "[2] Hendrycks, Dan, et al. \"Natural adversarial examples.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[3] Wang, Haohan, et al. \"Learning robust global representations by penalizing local predictive power.\" Advances in neural information processing systems 2019.", "other_strengths_and_weaknesses": "The artificial perturbations proposed are effective at degrading the performance of zero-shot models, loosely align to human perception (Figure 5) and have been shown to be fair because learnable (Table 1).\n\nOne limitation though is that the synthetic nature of the perturbations questions whether they represent real world occurrences to test actual failure cases of zero-shot models when deployed to the real world.\n\nIn this scope, a complementary approach similar to ImageNet-Adversarial but applied to LAION would be relevant.", "other_comments_or_suggestions": "The paper oversells the \"paradigm shift\" claim as the perturbations proposed and in the same vein as the ImageNet-C corruptions.", "questions_for_authors": "Have the authors considered constructing a OOD test set in a similar manner to the ImageNet-A dataset but for LAION to test real world failure cases ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "psLgsgw8fD", "reviewer": "Reviewer_xiHt", "summary": "This paper proposed a new dataset called LAION-C for evaluating image classification model robustness against out-of-distribution data. The LAION-C datasets select samples from ImageNet validation set, and added with 6 different types of synthetic distortions, each with 5 different levels of severity. A substantial evaluation was conducted to assess different commonly used neural network models performance on LAION-C dataset. A human review was also conducted to compare human and model performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nthank you for taking the time to provide such detailed and insightful feedback. We were delighted to read that you found our **experimental design [to be] in general thoughtful and comprehensive**, and appreciated the **substantial evaluation and insightful error analysis**. We have clarified the manuscript based on your suggestions, added citations, and scaled the dataset to a substantially larger version with 1k classes, for a total of 1.5M images.\n\n*Q: Motivation for choosing these 6 types of distortions.*\n\nOur distortions needed to fulfill the desiderata of 1. being exotic enough to have a low occurrence probability even in web-scale datasets and 2. testing relevant feature extraction capabilities of the models. We now include this motivation in Sec 2.1.\nTo solve the Stickers and Mosaic distortion, a model needs to be able to holistically integrate the image, instead of being led astray by local image cues induced by sub-images, which is notoriously difficult for DNNs [1].\nThe Glitch and Vertical Lines distortions are the most exotic and globally disruptive image transformations we could find, and destroy the texture cues that models rely on [1].\nThe Geometric Shapes distortion tests amodal completion, a staple of human visual processing even in infants [2,3]. They also change the color distribution of the image, which humans are robust to because we do not rely primarily on color for object recognition [4,5].\nThe Luminance Checkerboard distortion tests a model’s ability to adapt to local lighting conditions, an important skill of the human visual system [6].\n\n[1] Geirhos et al 2019\n\n[2] Kellmann and Spelke 1983\n\n[3] Nanay 2018\n\n[4] Tanaka and Presnell 1999\n\n[5] Biedermann and Ju 1988\n\n[6] Heeger 1992\n\n*Q: not much discussion or contrast with other OOD dataset*\n\nThanks for the suggestion - we have added a figure contrasting LAION-C with other OOD datasets: https://ibb.co/HfZmLkTd. This shows that LAION-C offers better resolution of model differences. Furthermore, in the camera-ready version we will substantially expand our discussion of other OOD datasets.\n\n*Q: I don't think the Table 1 result is sufficient.*\n\nThanks for the great suggestion. While our initial approach demonstrates learnable signals in LAION-C, your suggestion offers a stronger proof of concept. We now fine-tuned the same model backbone as in Tab.1 on a mixture of clean and distorted ImageNet-Train images and evaluated on both ImageNet-Val 16 class and LAION-C 16 class. The results illustrated here (https://ibb.co/rGmQk2rG) indeed show that *‘improvement of LAION-C accuracy can be achieved without regression on the clean dataset’*. We will update Sec. 3.3 accordingly with the results and full experiment setup to strengthen our argument.\n\n*Q: LAION-C only has 273 images*\n\nWe selected 273 images *per class*, yielding 4,368 base images and ~130k total images after corruption at different intensity levels. Nonetheless, we hear your point regarding broad coverage and scale. To address it, we extended our dataset to the full 50k IN-validation set, resulting in LAION-C-1k with 1.5 million images. The dataset is ready and has been submitted for review on Zenodo. While we are not including a direct link at this stage respecting the rebuttal policies, we will link the dataset in the camera-ready version. \n\n*Q: Why is it called LAION-C?*\n\nGreat question. We follow common practice in the OOD literature, where datasets are named w.r.t. the dataset for which they’re OOD. E.g., ImageNet-Sketch is designed to be OOD for ImageNet (even though none of the sketch images are part of ImageNet since that would defeat the purpose of being OOD). We now explain this motivation in the introduction. Applying corruption to LAION images directly is possible, but they don’t have class labels, thus cannot be used (without proxy/pseudo labels) for classification.\n\n*Q: [M]ore justification on the human performance is needed.*\n\nPlease see our response to ZcBa.\n\n*Q: LAION-C difficulty vs other protocols?*\n\nOur main results (Fig. 4) follow a cross-dataset evaluation protocol: The models were trained on web-scale datasets, fine-tuned on IN-1k but evaluated on LAION-C. In general, it is possible to apply any OOD evaluation protocol to LAION-C that one could also apply to other OOD-datasets, such as ImageNet-[A, R, C, Sketch]. \n\n*Q: Why are sticker and mosaic distortions more challenging?*\n\nGreat question. We introduced content-rich image tiles into the original image, to test the model's ability to process images holistically in spite of local distractions. These two distortions are built on simpler augmentations that mask or pixelate content, as illustrated in https://ibb.co/CpBvR2Y3. We observe an up to 35% drop on ViT-B32 compared to these easier distortions (https://ibb.co/hJj8pfQw), indicating that local signals pose a significant challenge. Fig.12 also supports this, as LLM-based models frequently direct attention toward the inserted tiles.", "claims_and_evidence": "1. One claim from the paper is that the state-of-the-art image recognition models were able to perform well on OOD datasets, while degrades substantially on LAION-C, which suggests LAION-C being a valuable dataset to test model performance for OOD datasets. This is supported by results in Figure 3. There is a consistent performance degradation for all the evaluated models from ImageNet-C to LAION-C. I think purely from performance point of view, LAION-C is a more challenging benchmark than ImageNet-C.\n2. Another claim from the paper is that models fine-tuned on LAION-C dataset can improve the performance that is comparable to clean dataset performance. This is supported by results in Table 1. This looks promising except for category Stickers and Mosaic Shape. It would be helpful to provide more insights on why these two types of distortions remain more challenging than other distortions. To be more specific, Stickers and Geometric Shape distortions both block information from clean image, but Geometric Shape distortion can be handled very well with re-training. Similarly, Mosaic and Vertical Lines obfuscate information, but Vertical Lines performance improves significantly with retraining. \n3. Another claim from the paper is that models can achieve comparable or superior performance in some OOD classes than human, while still struggle on other categories. For this claim, the evidence was provided in Figure 5. For this claim, more justification on the human performance is needed. We can see the best model performance has a much narrower 95% confidence interval compared to human. Is the best human performance based on aggregation of all the participates, or is it based on overall best individual performer? Given the small subject population and no-expertise required for participating the analysis, it would be more proper to consider claiming on average human performance rather than best human performance.\n4. Another claim from the paper is that LAION-C dataset is a challenging dataset but can potentially be solved. This is also supported by Table 1. I have some doubts on this claim as we cannot tell if the improved performance of model trained with LAION-C poses any regression on the clean data performance. In other words, it is unclear whether the model overfits to the distortion introduced in LAION-C or not. It would be helpful to show the same model (trained with LAION-C data) performance on clean and distorted images side-by-side.", "methods_and_evaluation_criteria": "1. The distortions were manually designed and programmatically generated. While it does pose challenge for current recognition models, I'm a bit skeptical about the practicality of these kind of distortion in real-world use case. It would be good to give motivation on why choosing these 6 types of distortions. \n2. The LAION-C uses clean images from ImageNet. If so, why it is called LAION-C? Shouldn't it be called another variant of ImageNet? More importantly, LAION dataset is known for its web-scale and diversity. But LAION-C is neither from web-scale images or diverse in terms of image classes. I think calling it LAION-C is misleading. \n3. The LAION-C dataset only have 273 clean original images. I understand that to have manageable human review experiment, we may not be able to use large scale dataset. But for a final release, I would suggest consider extending the number of images and number of classes further.", "theoretical_claims": "There is no theoretical claims.", "experimental_designs_or_analyses": "Centering around the proposed LAION-C dataset, the paper designed a series of experiments to answer following questions: 1) How challenging is the LAION-C dataset to modern image recognition models? 2) How human performs on LAION-C dataset? 3) Is there any difference in errors made by model and human? 4) Can LAION-C dataset be solved by recognition model? I found the experimental design is in general thoughtful and comprehensive. For example, on evaluating the extend of OOD of LAION-C, the authors utilized three different approaches including qualitative assessment, quantitative measurement of model performance, and quantitative measurement of difference between datasets. \n\nThe main concern I have though is all the analysis and discussion anchors on difference from ImageNet-C. This is probably originated from the fact that LAION-C use original images from ImageNet. There is not much discussion or contrast with other OOD dataset. Furthermore, model OOD work often use cross-dataset evaluation for evaluating model capability on handling OOD data. For example, in [1] where the model is trained on ImageNet-1K and tested on a bunch of other datasets. \n\n[1] Miyai, Atsuyuki, et al. \"Locoop: Few-shot out-of-distribution detection via prompt learning.\" Advances in Neural Information Processing Systems 36 (2023): 76298-76310.", "relation_to_broader_scientific_literature": "When discussing related work, the focus of this paper in on contrasting the LAION-C dataset and ImageNet based dataset. There is no discussion on other OOD datasets. In addition, there is no experiment design related to cross-dataset evaluation protocol, which is often adopted in OOD literature.", "essential_references_not_discussed": "Some of the references that can be added are \n\n[1] Miyai, Atsuyuki, et al. \"Locoop: Few-shot out-of-distribution detection via prompt learning.\" Advances in Neural Information Processing Systems 36 (2023): 76298-76310.\n[2] Yang, Jingkang, et al. \"Generalized out-of-distribution detection: A survey.\" International Journal of Computer Vision 132.12 (2024): 5635-5662.", "other_strengths_and_weaknesses": "[Other strengths]\n+ A variety of model image recognition models were evaluated and compared for performance on LAION-C\n+ The error consistency analysis between human and models is insightful to show different failure modes of human vision and computer vision.\n+ The appendix provides a comprehensive discussion on what is LAION-C and how it is constructed using the template format.\n\n[Other weaknesses]\n- There is no justification from practical or theoretical point of view why using the 6 proposed distortions. The proposal of distortion types seem a bit arbitrary. It is unclear about the implication of a model performing well on LAION-C dataset. \n- There is limited novelty in terms of dataset construction in the three aspects: 1) the clean images are selected from existing ImageNet dataset; 2) the way of constructing OOD samples is based on existing practice by artificially adding obstruction to or alternating the image, which has been done in ImageNet-C. The only difference is the type of distortion; 3) there is no new proposals on the evaluation of OOD performance, which is still based on multi-class classification accuracy..", "other_comments_or_suggestions": "Typo in supplementary materials: 131.040 should be 131,040.", "questions_for_authors": "1. I do not fully understand why this proposed dataset is called LAION-C while all its original clean images come from ImageNet. In addition, the number of original images is only 273 from 16 classes, even though there are a total of 131,040 images accounting for all distortion variations. So I don't think it resembles the diverse collection and scale of LAION dataset. It would be helpful if the authors can clarify the rationale of naming the dataset as LAION-C instead of another variant of ImageNet. It seems the same kind of distortion can be applied to LAION images as well, in which case the resulting dataset can also be called 'LAION-C'. Why not?\n2. On the justification of LAION-C can be solved, I don't think the Table 1 result is sufficient. A critical question is whether the improvement of LAION-C accuracy can be achieved without regression on clean dataset. It would be good to evaluate the same model performance on clean dataset (see more discussion in 'Claims And Evidence'). If there is no regression on clean data, that means model performance on LAION-C can simply be improved by adding distortion as an augmentation. It is indeed promising to solve LAION-C. Otherwise, the challenge of achieving robustness on unseen distortions remain unsolved and more evidence is needed on whether LAION-C can be solved. \n3. OOD evaluation in recent literatures are typically based on using distinct training and testing datasets such as training under ImageNet and testing on iNaturalist. There is no discussion or comparison of LAION-C with this popular OOD evaluation protocol. It may be that model performance on ImageNet-C becomes more saturated, but I'm not sure it is the case that such cross-dataset evaluation also becomes saturated. In that sense, the claim on modern image recognition models saturated on current benchmark is not well supported. Could the authors clarify what is the level of difficulty of evaluating on LAION-C compared to other cross-dataset evaluation protocol?\n4. Related to point 3, what is the main novelty of constructing LAION-C? \n5. Related to point 1 and 2, what is the significance of solving LAION-C? Can we draw a better conclusion that if a model does a better job on LAION-C, it has better robustness against OOD data regardless of its performance on other OOD evaluation benchmark? If not, what other insights we can gain from the performance on LAION-C dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nj6pdFQNeY", "reviewer": "Reviewer_ZcBa", "summary": "This paper presents a novel dataset, designated as LAION-C, where the letter C denotes \"corrupted.\" This dataset bears similarities to ImageNet and ImageNet-C. \nThe dataset under consideration contains six corruptions (Mosaic, Glitched, Vertical Lines, Stickers, Geometric Shapes, Luminance Checkerboard) and 16 superclasses.\nRather than focusing on natural corruptions, the objective is to create a highly synthetic corrupted dataset. \nThe evaluation process entails the psychophysical observation of 19 participants.\n\n## update after rebuttal\nNone", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your favourable review. We appreciate that you find the utilization of LAION-C **logical** because it **poses a new challenge for OOD-evaluation**, and **value the human experiment**. Prompted by your feedback, we will update the manuscript to include a section motivating our artificial corruptions better, and publish a version of our benchmark that consists of 1,000 class labels (LAION-C-1K). Thank you for alerting us to the existing nomenclature regarding the different types of distribution shifts (semantic vs covariate shifts); we have updated the manuscript accordingly. Our shift is indeed of the covariate shift kind.\n\n\n*Q: “Did you take into account that the ImageNet has .png (validation set) images and the Laion-C has .jpg compressed images?”*\n\nA: Thank you for raising this point. We double-checked that ImageNet images are JPEG-compressed in both training and validation sets. Since LAION-C base images were taken from the ImageNet validation set, analogous to ImageNet-C, this design choice matches other benchmarks. We therefore also use JPEG-compression, setting parameters which give close to lossless compression.\n\n*Q: “Are 19 evaluators a good number?”*\n\nA: Valid question. Yes, 19 evaluators are a sufficient and typical number for this type of study: Since each human observer is exposed to 2 corruptions, every corruption is seen by at least 6 different humans, assuring good coverage. For reference, [1] used only 4 humans per corruption for most of their corruptions. In psychophysics, this type of design is called a “small N design” (fewer participants that see many images, [3]). More quantitatively, we see in Fig. 10 that the 95% confidence intervals surrounding our estimate of the average human performance are quite small (+- 4.61%). Statistically speaking, it is therefore very unlikely that adding more human observers would change any of the  results or interpretations. We will add a detailed discussion of this to the Appendix. Since beating the best human is more difficult than beating the average human, we analyze peak performance rather than average performance in Fig. 5 and it is impressive to see that good models beat every single human. The confidence interval is smaller for the models because they were evaluated on all images, while each human only saw a subset of images for practical reasons. In the Appendix, we will now include a comparison to both average and peak human performance.\"\n\n*Q: “Why only 16 classes?”*\n\nA: We used 16 classes to enable a comparison to human observers in line with earlier work [1, 2], because humans cannot be expected to reliably classify images into too many categories, simply due to pragmatic limitations (size of the response icons, time per trial etc.) Prompted by your suggestions, we have decided to also provide a version of our dataset consisting of all 1,000 ImageNet classes, which we call LAION-C-1k. The dataset is ready and has been submitted for review on Zenodo. While we are not including a direct link at this stage respecting the rebuttal policies, we will link the dataset in the camera-ready version. \n\n*Q: “How should I use the different intensity levels?”*\n\nA: We included different intensity levels so that practitioners would be able to see the decline in model performance and obtain a more fine-grained evaluation of robustness. This is done by some other OOD datasets as well, like model-vs-human [1] and ImageNet-C. The protocol is to evaluate a new model on all levels of our benchmark and report the average across levels for the benchmark result (we’ll make sure to make this clear in the manuscript and github to ensure comparability). Additionally, users are free to report performance curves like we do in e.g. Fig. 5.\n\n[1]: Partial success in closing the gap between human and machine vision (Geirhos et al., 2021)\n\n[2]: Generalisation in humans and deep neural networks (Geirhos et al., 2018)\n\n[3] Small is beautiful: In defense of the small-N design (Smith and Little, 2018)", "claims_and_evidence": "The correlation between model size and robustness is positive; that is to say, the larger the model, the more robust it becomes. \nThis assertion is supported by empirical evidence from experiments conducted on vision foundation models, which involved the utilization of 19 human evaluators. However, it is noteworthy that the performance of these human evaluators frequently surpasses that of the models.", "methods_and_evaluation_criteria": "The provided dataset is a benchmark for out-of-distribution (OOD) detection. It has been observed that the accuracy of the models decreases to a greater extent than in ImageNet-C; therefore, the utilization of this dataset is logical.", "theoretical_claims": "None", "experimental_designs_or_analyses": "The primary concern pertains to the composition of the evaluator cohort, which consists of 19 individuals.\nA comparison between ImageNet-C and LAION-C demonstrated the significance of this factor.", "relation_to_broader_scientific_literature": "OOD detectors have become better and is still a challenge.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths\n- This dataset can be a new challenge in OOD detection.\n- Model often outperform humans.\n- Their results show that vision foundation models evaluated on ImageNet-C are robust. \nBut evaluated on LION-C they show less accuracy.\n- Dataset was manually filtered. To ensure that only one object is on the image. \n- Mechanical writing.\n\nWeaknesses\n- Only 16 classes: Some OOD detectors could be class dependant, and fewer classes could be easier for them.\n- The explanation why artificial corruption are important. \n- there is no categorization of distribution shift. Covariate? [1]\n\n[1] https://arxiv.org/pdf/2501.18463v2", "other_comments_or_suggestions": "None", "questions_for_authors": "- Did you take into account that the ImageNet has .png (validation set) images and the Laion-C has .jpg compressed images? \n- Are 19 evaluators a good number? \n- Why only 16 classes? Isn't that easier for some OOD detectors? The hierarchical 1000 classes of ImageNet would be more difficult if the detector takes the class probabilities into account? \n- There are different intensity levels. The ImageNet-C does not have this. If I want to use this dataset, how should I use the different intensity levels?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}]

Author rebuttal:

Meta-review: This paper received two positive reviews, and one weak reject where the reviewer, xiHt, acknowledged that the rebuttal addressed most of his concerns. The AC agrees that the rebuttals were convincing, and the paper should be accepted. It introduces a large, new dataset for OOD detection that modernizes the field beyond the typical ImageNet benchmarks by addressing scaling limitations. The human study of OOD detection is also a useful contribution.
Decision: Accept (poster)