A Decade's Battle on Dataset Bias: Are We There Yet?
Authors: Zhuang Liu
Kaiming He
Venue: ICLR 2025 Oral (2025)
Keywords: Vision datasets
Dataset bias
Deep learning
Primary area: datasets and benchmarks
TL;DR: Modern large-scale vision datasets that are supposedly very general and diverse, are in fact still very biased

Abstract: We revisit the ``dataset classification'' experiment suggested by Torralba & Efros (2011) a decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, we observe that modern neural networks can achieve excellent accuracy in classifying which dataset an image is from: e.g., we report 84.7% accuracy on held-out validation data for the three-way classification problem consisting of the YFCC, CC, and DataComp datasets. Our further experiments show that such a dataset classifier could learn semantic features that are generalizable and transferable, which cannot be explained by memorization. We hope our discovery will inspire the community to rethink issues involving dataset bias.

Published as a conference paper at ICLR 2025
A DECADE'S BATTLE ON DATASET BIAS: ARE WE THERE YET?
Zhuang Liu Kaiming He Meta AI Research, FAIR
ABSTRACT
We revisit the "dataset classification" experiment suggested by Torralba & Efros (2011) decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, observe that modern neural networks can achieve excellent accuracy in classify- ing which dataset an image is from: e.g., we report 84.7% accuracy on held-out validation data for the three-way classification problem consisting of the YFCC, CC, and DataComp datasets. Our further experiments show that such a dataset classifier could learn semantic features that generalizable and transferable, which cannot be explained by memorization. We hope our discovery will inspire the community rethink issues involving dataset bias.
INTRODUCTION
In 2011, Torralba & Efros (2011) called for battle against dataset bias in the community, right before the dawn of the deep learning revolution (Krizhevsky et al., 2012). They introduced the "Name That Dataser" experiment, where images are sampled from a dataset and a model is trained on the union of these images classify which dataset an image is taken. Remarkably, datasets at that time could be classified with high accuracy. They also found that model trained on one dataset can only perform well on that dataset but fails to generalize to others.
In response to this, over the decade that followed, progress on building diverse, large-scale, com- prehensive, and hopefully less biased datasets (Lin al., 2014; Russakovsky al., 2015; Thomee et al., 2016; Kuznetsova et al., 2020; Schuhmann et al., 2022) has been an engine powering the deep learning revolution, especially in the pre-training era. In parallel, advances in algorithms, particularly neural network architectures, have achieved unprecedented levels of ability on discovering concepts, abstractions, and patterns-inclun
bias-from data.
In this work, we take renewed "unbiased look at dataset bias" (Torralba & Efros, 2011) after the decade-long battle. Our study is driven by the tension between building less biased datasets versus developing more capable models the latter was less prominent at the time of Torralba & Efros (2011). While efforts to reduce bias in data may lead to progress, the development of advanced models could better exploit dataset bias and thus counteract the promise.
Our study is based on a fabricated task we call dataset classification, which is the "Name That Dataset" experiment designed in Torralba & Efros (2011) (Figure 1). The datasets we experiment with are presumably among the most diverse, largest, and uncurated datasets in the wild, collected from the Internet. For example, typical combination we study, referred as "YCD", consists of images from YFCC (Thomee et al., 2016), 6), CC (Changpinyo et al., 2021), and DataComp (Gadre al., 2023) and presents a 3-way dataset classification problem.
To our (and many of our initial readers") surprise, modern neural networks can achieve excellent accuracy on such a dataset classification task. Trained in the aforementioned YCD set that is challenging for human beings (Figure 1), a model can achieve >84% classification accuracy on the held-out validation data, vs. 33.3% of chance-level guess. This observation is highly robust, over a large variety of dataset combinations and across different generations of architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016; Dosovitskiy et al., 2021; Liu et al., 2022), with very high accuracy (e.g., over 80%) achieved in most cases.
Work done Meta; now MIT. Code: gifuuh.omm/luzhungg 13/bias

---

Published as a conference paper at ICLR 2025
10
12
15
16
17
18 19
20
21
Figure 1: The "Name That Dataset" game (Torralba & Efros, 2011) in 2024: These images are sampled from three modern datasets: YFCC (Thomee et al., 2016), CC (Changpinyo et al., 2021), and DataComp (Gadre al., 2023). Can you specify which dataset each image is from? While these datasets appear to be less biased, we discover that neural networks can easily accomplish this "dataset classification" " task with surprisingly high accuracy on the held-out validation set.
Answer: YFCC: 1, 4. 7, 10, 16, 19; CC: 2, 14, 17, DataComp: 6, 9, 12, 15, 18,
For such a dataset classification task, we have a series of observations that are analogous to those observed in semantic classification tasks (e.g., object classification). For example, we observe that training the dataset classifier on more samples, or using stronger data augmentation. can improve accuracy held-out validation data, even though the training task becomes harder. This is similar to the generalization behavior in semantic classification tasks. This behavior suggests that the neural network attempts discover dataset-specific patterns-a form of bias-to solve the dataset classification task. Further experiments suggest that the representations learned by classifying datasets carry some semantic information that transferrable image classification tasks.
As a comparison, if the samples of different datasets were unbiasedly drawn from the same distribution, the model should not discover any dataset-specific bias. To check this, we study a pseudo-datased classification task, in which the different "datasets" are uniformly sampled from a single dataset. We observe that this classification task quickly becomes intractable, as the only way for the classifier to approach this task is to memorize every single instance and its subset identity. As result, increasing the number of samples, or using stronger data augmentation, makes memorization more difficult or intractable in experiments. No transferability is observed. These behaviors are strikingly contrary to those of the real dataset classification task.
More surprisingly to us. observe that self supervupd learning models are also highly capable of capturing certain bias among different datasets. Specifically, we pre-train a self-supervised model on the union of different datasets, without using any dataset identity as the labels. Then with the pre-trained representations frozen, we train a linear classifier for the dataset classification task. Although this linear layer the only layer that is tunable by the dataset identity labels, the model can still achieve a high accuracy (e.g., 78%) for dataset classification. This transfer learning behavior resembles the behaviors of typical self-supervised learning methods (e.g., for image classification).
In summary, we report that modern neural networks are surprisingly capable of discovering hidden bias from different datasets. This observation is true even for modern datasets that are very large, diverse, less curated, and presumably less biased. The neural networks can solve this task by discovering generalizable patterns (i.e., generalizable from training data to validation data, or to downstream tasks), exhibiting behaviors analogous to those observed in semantic classification tasks. Comparing with the game of "Name That Dataset" in Torralba & Efros (201 decade ago, this game even becomes way easier given today's capable neural networks. In this sense, the issue involving dataset bias has not been relieved. There is still a question about how representative our current pre-trate seetrranning datasets are of the real world, and furthermore, how much more generalizable models could become by building more diverse and less biased training datasets. We hope our discovery will stimulate discussions in the community regarding dataset bias in this new era.
2

---

Published as a conference paper at ICLR 2025
2 A BRIEF HISTORY OF DATASETS
Pre-dataset Eras. The concept of "datasets" did not emerge directly out of the box in the history of computer vision research. Before the advent of computers (e.g.. see Helmholtz's 'ss book of the 1860s (Von Helmholtz, 1867)), scientists had already recognized the necessity of "test samples", often called "stimuli" back then, to examine their computational models about the human vision system. The stimuli often consisted of synthesized patterns, such as lines, stripes, and blobs. The practice of using synthesized patterns was followed in early works on computer vision.
Immediately after the introduction of devices for digitizing photos, researchers were able to validate and justify their algorithms on one or very few real-world images (Roberts, 1963). For example, the Cameraman image (Schreiber, 1978) has been serving as standard test image for image processing research since 1978. The concept of using data (which was not popularly referred to as "datasets") to evaluate computer vision algorithms was gradually formed by the community.
Datasets for Task Definition. With the introduction of machine learning methods into the computer vision community, the concept of "datasets" became clearer. In addition to the data for the validation purpose, the application of machine learning introduced the concept of training data, from which the algorithms can optimize their model parameters.
As such, the training data and validation data put together inherently define a task that is of interest. For example, the MNIST dataset (LeCun et al., 1998) defines a 10-digit classification task; the Caltech- 101 dataset (Fei-Fei et al., 2004) defines an image classification task of 101 object categories; the PASCAL VOC suite of datasets (Everingham et al., 2010) define a family of classification, detection, and segmentation tasks of 20 object categories.
To incentivize more capable algorithms, more challenging tasks were defined. The most notable example of this kind, in today's 's context, the ImageNet dataset (Deng et al., 2009). ImageNet has over one million images defined with 1000 classes (many of them being fine-grained animal species), which nontrivial even for normal human beings to recognize (Karpathy, 2014). At the time when ImageNet was proposed, algorithms for solving this task appeared to be cumbersome e.g., the organizers provided SIFT features (Lowe, 2004) pre-computed to facilitate studying this problem, and typical methods back then may train 1000 SVM classifiers, which in itself is a nontrivial problem (Vedaldi & Zisserman, 2012). Hypothetically. if ImageNet was to remain as a task on its own, like many previous popular datasets, we wouldn't 't be able to witness the deep learning revolution.
But a paradigm shift awaited.
Datasets for Representation Learning. Right after the deep learning revolution in 2012 (Krizhevsky et al., 2012). the community soon discovered that the neural network representations learned on large-scale datasets like ImageNet are transferrable (Donahue et al., 2014; Girshick et al., 2014; Yosinski et al., 2014). The discovery brought in a paradigm shift in computer vision: it became common practice to pre-train representations on ImageNet and transfer them to downstream tasks.
As such, the ImageNet dataset was no longer a task of its own: it became a pinhole of the universal visual world that we want to represent. Thus, the used-to-be cumbersome aspects became advantages of this dataset: a larger number of images and more diversified categories than most other datasets at that time, and empirically these properties turned out to be important for learning good representations.
Encouraged by ImageNet's 's enormous success, the community began to pursue more general and ideally universal visual representations. Tremendous effort has been paid on building larger, more diversified, and hopefully less biased datasets. Examples include YFCC100M (Thomee et al., 2016), CC12M (Changpinyo et al., 2021), and DataComp- 1B (Gadre et al., 2023) the main datasets we study in this paper-among many others (Sun et al., 2017: Desai et al., 2021; Srinivasan et al., 2021; Schuhmann et al., 2022). It is intriguing to notice that the building of these datasets does not always define task of interest to solve; actually, many of these large-scale datasets do not provide a split of training / validation sets. It is with the goal of pre-training in mind that these datasets were built.
3
ON DATASET BIAS
Given the increasing importance of datasets, the bias introduced by datasets has drawn the commu- nity's attention. Torralba & Efros (201) presented the dataset classification problem and examined
3

---

Published as a conference paper at ICLR 2025
dataset bias in the context of hand-crafted features with SVM classifiers. Tommasi al. (2015) studied the dataset classification problem using neural networks, specifically focusing on linear classifiers with pre-trained ConvNet features (Donahue et al., 2014). The datasets they studied are smaller in scale and simpler comparing with today's 's web-scale data.
The concept of classifying different datasets has been further developed in domain adaption meth- ods (Tzeng et al., 2014; Ganin et al., 2016). These methods learn classifiers to adversarially distinguish features from different domains, where each domain can be thought of as a dataset. The problems studied by these methods are known to have significant domain gaps. On the contrary, the datasets we study are presumably less distinguishable, at least for human beings.
Another direction on studying dataset bias is to replicate the collection process of a dataset and examine the replicated data. ImageNetV2 (Recht et al., 2019) replicated the ImageNet validation set's protocol. It observed that this replicated data still clearly exhibits bias as reflected by accuracy degradation. The bias is further analyzed in (Engstrom et al., 2020).
Many benchmarks (Hendrycks & Dietterich, 2018: Zendel et al., 2018; Koh et al., 2021; Hendrycks et al., 2021) have been created for testing models' generalization under various forms of biases, such as common corruptions and hazardous conditions. There is also a rich line of work on mitigating dataset bias. Training on multiple datasets (Lambert et al., 2020; Nguyen et al., 2022) can potentially mitigate dataset bias. Methods that adapt models to data with different biases at test time (Sun et al., 2020; Wang et al., 2021) have also gained popularity recently.
Different Notions of Bias. It is worth noting that this study's focus is the bias among multiple datasets (hence "dataset" bias, instead of "data" bias). This mostly concerns the proper coverage of concepts and objects, or in other words, how representative the dataset is for the real world. It is not to be confused with another common notion of bias in data-sooee and stereotypical bias. This notion concerns more on algorithmic fairness (Mitchell et al., 2021) and could be found within single dataset, e.g., gender or race bias. These two notions are related but emphasize different aspects. For example, simple dataset of indoor furnitures is mostly free of social bias, but is extremely biased in terms of representativeness of the world.
Addressing social bias in data is an active area of research. Several well-known datasets have been identified with biases in demographics (Buolamwini & Gebru, 2018; Yang et al., 2020) and geography (Shankar et al., 2017). They also contain harmful socictal stereotypes (van Miltenburg, 2016; Prabhu & Birhane, 2021; Birhane et al., 2021; Zhao et al., 2021). Addressing these biases is critical for fairness and ethical considerations. Tools like REVISE (Wang al., 2022) and Know Your Data (Google People + AI Research, 2021) offer automatic analysis for potential bias in datasets. Debiasing approaches, such as adversarial learning (Zhang et al., 201 8a) and domain-independent training (Wang et al., 2020), have also shown promise in reducing the effects of dataset bias.
4 DATASET CLASSIFICATION
The dataset classification task (Torralba & Efros, 2011) is defined like an image classification task, but each dataset forms its own class. It creates an N-way classification problem where N is the number of datasets. The accuracy is evaluated on held-out validation set sampled from these datasets.
4.1 ON THE DATASETS WE USE
We intentionally choose the datasets that can make the dataset classification task challenging. We choose our datasets based on the following considerations: (1) They are large in scale. Smaller datasets might have a narrower range of concepts covered, and they may not have enough training images for dataset classification. (2) They are general and diversified. We avoid datasets that are about specific scenario (e.g., cities (Cordts et al., 2016), scenes (Zhou et al., 2017)) or a specific meta-category of objects (e.g., flowers (Nilsback & Zisserman, 2008), pets (Parkhi et al., 2012)). (3) They are collected as with the intention of pre-training generalizable representations, or have been used with this intention. We emphasize the difference between the "pre-training" datasets and "benchmark" datasets here, as it is more accepted that the evaluation benchmark datasets are often unique and biased (Raji et al., 2021; Koch et al., 2021). Based on these criteria, we choose the datasets listed in Table 1.
4

---

Published as a conference paper at ICLR 2025
dataset
description
YFCC (Thomee al., 2016)
100M Flickr images
CC (Changpinyo et al., 2021)
12M Internet image-texi pairs
DataComp (Gadre et al., 2023)
1B image-text pairs from Common Crawl
WTT (Srinivasan et al., 2021)
1.55M Wikipedia images-text pairs
LAION (Schuhmann et al., 2022)
2B image-text pairs from Common Crawl
ImageNet (Deng et al., 2009)
14M images from search engines
Table 1: Datasets used in our experiments.
Although these datasets are supposedly more diverse, there are still differences in their collection processes that potentially contribute their individual biases. For example, their sources are different: Flickr is website where users upload and share photos, Wikipedia is website focused on knowledge and information, Common Crawl is an organization that crawls the web data, and the broader Internet involves more general range of content than these specific websites. Moreover, different levels of curation have been involved in the data collection process: e.g., LAION was collected by reverse-engineering the CLIP model (Radford al., 2021) and reproducing its zero-shot accuracy (Schuhmann et al., 2022).
Despite awareness of these potential biases, neural network's excellent ability capture them beyond our expectation. In particular, we note that we evaluate network's s dataset classification accuracy by applying each validation image individually, which ensures that the network has no opportunity to exploit the underlying statistics of several images.
4.2 MAIN OBSERVATION
We observe surprisingly high accuracy achieved by neural networks in this dataset classification task. This observation is robust across different settings. By default, we randomly sample 1M and 10K images from each dataset training and validation sets, respectively. We train ConvNeXt-T model (Liu et al., 2022) following common practice of supervised training (implementation details in Appendix B).
YFCC
CC
DataComp
WIT
LAION
ImageNet
accuracy
84.7
83.9
85.0
92.7
85.8
72.1
90.2
86.6
86.7
91.9
83.6
62.8
82.8
84.3
91.3
84.1
71.5
88.9
68.2
90.7
84.7
79.1
67.4
69.2
Table 2: Dataset classification yields high accuracy in all combinations. Top panel: all 20 combinations that involve datasets out of all Bottom panel: combinations with 3, 4, 5, or datasets. All results with 1M training images sampled from each dataset.
5

---

Published as a conference paper at ICLR 2025
We observe the following behaviors in our experiments:
High accuracy is observed across dataset combinations. In Table 2 (top panel), we enumerate all 20 (C²) possible combinations of choosing out of the datasets listed in Table 1. summary. in all cases, the network achieves 662 dataset classification accuracy; and in 16 out of all 20 combinations, it even achieves >80% accuracy. In the combination of YFCC, CC, and ImageNet, achieves the highest accuracy of 92.7 %. Note that the chance-level guess gives 33.3% accuracy.
In Table (bottom panel), we study combinations involving 3, 4, 5, and all datasets. As expected, using more datasets leads to more difficult task, reflected by the decreasing accuracy. However, the network still achieves 69.2% accuracy when all 6 datasets are included. The confusion matrix of the 6-way classifier can be found in Appendix C.
High accuracy is observed across model architectures. Table 3 shows the results on YCD using different generations models: AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), VIT (Dosovitskiy et al., 2021), and ConvNeXt (Liu et al., 2022).
We observe that all archilectures can solve the task excellently: 4 out of the 5 networks achieve excellent accuracy >> 80%, and even the clas- sical AlexNet achieves a strong result of 77.8%.
model
accuracy
AlexNet
77.8
VGG-16
83.5
Re
83,8
This result shows the neural networks are ex- tremely good at capturing dataset biases, regard- less of their concrete architectures. There has been significant progress in network architec- ture design after the AlexNet paper, including normalization layers (loffe Szegedy, 2015;
ViT-S
82.4
ConvNeXi-T
84.7
Table 3: Different model architectures all achieve high accuracy. Results are on the YCD combination with 1M images each.
Ba et al., 2016), residual connections (He et al., 2016), self-attention (Vaswani et al., 2017; Dosovit- skiy et al., 2021). The "inductive bias" in network architectures can also be different (Dosovitskiy et al., 2021). Nevertheless, none of them appears to be indispensable for dataset classification (e.g.. VGG (Simonyan Zisserman, 2015) has none of these components): the ability to capture dataset bias may be inherent in deep neural networks, rather than enabled by specific components.
High accuracy is observed across different model sizes. By default, we use ConvNeXi- Tiny (27M parameters) (Liu al., 2022). The term "Tiny" is with reference to the modern definition of VIT sizes (Touvron et al., 2021; Dosovitskiy et al., 2021) and is comparable to ResNet-50 (25M) (He et al., 2016). In Figure 2, we report results of models with different sizes by varying widths and depth.
To our further surprise, even very small models can achieve strong accuracy for the dataset classi- fication task. ConvNe with as few as 7K parameters (3/10000 of ResNet-50) achieves 72.4% accuracy classifying YCD. This suggests that neural networks' structures are very effective in learning the underlying dataset biases. Dataset classification can be done without a massive number of parameters, which is often credited for deep learning's ' success in conventional visual recognition.
accuracy
100
80
72.4
76.0
79.7
83,083.6 84.7 85.0
accuracy
100
61.4
68.0
72.4
77.8
80.1
77.8
84,7 7
60
60
69.0
52.8
60.1
40
40
ConvNeXi AlexNet
20
ConvNeXt variants
20
7K
120K 470K model parameters
3M 8M 27M 87M
100
IK
10K
100K
IM
training images per dataset
Figure 2: Models of different sizes all achieve very high accuracy, while they can still be sub- stantially smaller than the sizes of typical mod- ern networks. Here the models are variants of ConvNeXt (Liu et al., 2022), whose "Tiny" size has 27M parameters. Results are on YCD com- bination with IM training images from each set.
Figure 3: Dataset classification accuracy in- creases with the number training images. This behavior suggests that the model is learn- ing certain patterns that are generalizable, which resembles the behavior observed in typical se- mantic classification tasks. Results are on YCD, with each model trained for the same iterations.

---

Published as a conference paper at ICLR 2025
We also observe that larger models get increasingly better, although the return becomes diminishing. This is consistent with observations on conventional visual recognition tasks. Moreover, we have not observed overfitting behaviors the extent of the model sizes and dataset scales we have studied. This implies that there may exist generalizable patterns that help the models determine dataset identities and the model is not trying memorize the training data. More investigations generalization and memorization are presented next.
Dataset classification accuracy benefits from more training data. We vary the number of training images for YCD classification and present results in Figure 3.
Intriguingly, models trained with more data achieve higher validation accuracy. This trend is consistently observed in both the modern Ne and the classical AlexNet. While this behavior appears be natural in semantic classification tasks, we remark that this not necessarily true in dataset classífication: in fact, if the models were focusing on memorizing the training data, their generalization performance on the validation data might decrease. The observed behavior-i.e, more training data improves validation accuracy- patterns that are generalizable to unseen data, rather than memorizing or overfitting the training data.
that the model is learning certain semantic
Dataset classification accuracy benefits from data augmentation. Data augmentation (Krizhevsky et al., 2012) is expected to have similar effects as increasing the dataset size (which is the rationale behind its naming). Our default training setting uses random cropping (Szegedy al., 2015), RandAug (Cubuk et al., 2020), MixUp (Zhang et al., 2018b), 8b), and CutMix (Yun et al., 2019) as data augmentations. Table shows the results of using reduced no data augmentations.
augmentation training images per dataset no aug
10K
100K
1M
43.2
71.9
76.8
w/ RandCrop
66.1
74.5
84.2
w/ RandCrop, RandAug
70.2
78.0
85.0
w/ RandCrop. RandAug. MixUp/ CuiMix
72.4
84.7
Table 4: Data augmentation improves dataset classification accuracy, similar to the behavior of semantic classification tasks. Results are on the YCD combination.
Adding data augmentation makes it more difficult to memorize the training images, while we observe that using stronger data augmentation consistently improves the dataset classification accuracy. This behavior remains largely consistent regardless the number of training images per dataset. Again, this behavior mirrors that observed in semantic classification tasks, suggesting that dataset classification is approached not through memorization, but by learning patterns that are generalizable from the training set the unseen validation set.
Summary. In sum, we have observed that neural networks are highly capable of solving the dataset classification task with good accuracy. This observation holds true across a variety of conditions, including different combinations of datasets, various model architectures, different model sizes, dataset sizes, and data augmentation strategies. In comparison, we report human performance on similar task in Appendix A.
ANALYSIS
In this section, we analyze the model behaviors in different modified versions involving the dataset classification task. This reveals more intriguing properties of neural networks for dataset classification.
5.1 LOW-LEVEL SIGNATURES?
There is a possibility that the high accuracy is simply due to low-level signatures, which are less noticeable to humans but are easily identifiable by neural networks. Potential signatures could involve JPEG compression artifacts (e.g.. different datasets may have different compression quality factors) and color quantization artifacts (e.g., colors are trimmed or quantized depending on the individual dataset). We design of experiments that help preclude this possibility.
7

---

Published as a conference paper at ICLR 2025
original
color jitter
noise
blur
low
Hap
Huppy
Figure 4: Different corruptions for suppressing low-level signatures. We apply certain type of corruption to both the training and validation sets, on which we train and evaluate our model.
Specifically, we apply certain type of image corruption to both the training and validation sets, on which we train and evaluate our model. In other words, we perform the dataset classification task corrupted data." We consider four types of image corruption: (i) color jittering (Krizhevsky et al., 2012), (ii) adding Gaussian noise with a fixed standard deviation; (iii) blurring the image by fixed-size Gaussian kernel; and (iv) reducing the image resolution. Figure 4 shows examples for each corruption. Note that we apply one type of corruption each time.
Table 5 shows the dataset classification results for each image corruption. As expected, corruption reduces the classification accuracy, as both training and validation sets are affected. Despite degrada- tion, strong classification accuracy can still be achieved, especially when the degree of corruption is weaker. Introducing these different types of corruption should effectively disrupt low-level signatures, such JPEG or color quantization artifacts. The results imply that the models attempt to solve the dataset classification task beyond using low-level biases.
corruption (on train+val)
accuracy
imgs per set
w/o aug
w/ aug
none
84.7
100
100.0
100.0
color jittering (strength: 1.0)
81.1
1K
100.0
100.0
color jittering (strength: 2.0)
80.2
10K
100.0
Gaussian noise (std: 0.2)
77.3
100K
fail
fail
Gaussian noise (std: 0.3)
75.1
Gaussian blur (radius: 3)
80.9
Table 6: Training accuracy on a pseudo- dataset classification task. Here create 3 pseudo-datasets, all which are sampled with- out replacement from the same souree dataset (YFCC). This training task is more difficult for the network solve given ages and/or stronger data augmentation. Val- idation accuracy is ~33% as no transferrable pattern is learned.
Gaussian blur (radius:
78.1
low resolution (64x 64)
78.4
low resolution (32x32)
68.4
Table 5: High accuracy are achieved on dif- ferent corrupted versions of the dataset clas- sification task. This suggests that low-level signature is not a main responsible factor. Re- sults are on the YCD combination.
training im-
5.2 MEMORIZATION OR GENERALIZATION?
In Sec. 4.2, we have shown that the models learned for dataset classification behave like those learned for semantic classification tasks (Figure 3 and Table 4), since they exhibit generalization behaviors. This behavior is in sharp contrast with memorization behavior, as we discuss in the next comparison.
'This is different from data augmentation, which applies random image corruption the training data.
8

---

Published as a conference paper at ICLR 2025
We consider assudo-datased classification task. In this scenario, manually create multiple pscudo-datasets, all of which are sampled without replacement from the same source dataset. We expect this process to give multiple pseudo datusessssssss that are truly unbiased.
Table reports the training accuracy of a model trained for this pseudo- dataset classification task, using different numbers of training images per set, without vs. with data augmentation. When the task is relatively simple, the model achieves 100% training accuracy; however, when the task becomes more difficult (more training images or stronger augmentation), the model fails converge, as reflected by unstable, non-decreasing loss curves.
This phenomenon implies that the model attempts memorize individual images and their labels to accomplish this pseudo-dataset classification task. Because the images in these pseudo-datasets are unbiased, there should be no shared patterns that can be discovered discriminate these different sets. As result, the model is forced to memorize the images and their random labels, similar to the scenario in Zhang al. (2017). But memorization becomes more difficult when given more training images or stronger augmentation, which fails the training process after certain point.
This phenomenon is unlike what we have observed in our real dataset classification task (Figure and Table 4). This again suggests that the model attempts to capture shared, generalizable patterns in the real dataset classification task.
Although it may seem evident, we note that the model trained for the pseudo-datase classification task does not generalize to validation data (which is held out and sampled from each pseudo-dataset), Even when the training accuracy is 100%, we report a chance-level accuracy of <33% in the validation set.
5.3 SELF-SUPERVISEN LEARNING
Thus far, all our dataset classification results are presented under a fully-supervised protocol: the models are trained end-to-end with full supervision. Next, explore self-supervised protocol, following the common protocol used for semantic classification tasks in self-supervised learning.
Formally, we pre-train self supervised learning model MAE (He al., 2022) without using any labels. Then we freeze the features extracted from this pre- pre-trained model, and train a linear classifier using supervision for the dataset classification task. This referred as the linear probing protocol. We note that in this protocol, only the linear classifier layer is tunable under the supervision of the dataset classification labels. Linear probing presents more challenging scenario.
Table 7 shows the results under the self-supervised protocol. Even with MAE pre-trained on standard ImageNet (which involves no YCD images), the model achieves 76.2% linear probing accuracy for dataset classification. In this case, only the linear classifier layer is exposed to the classification data.
Using MAE pre-trained the same YCD training data, the model achieves higher accuracy of 78.4% in linear probing. Note that although this MAE is pre-trained on the same target data, it has no prior knowledge that the goal is for dataset classification. Nevertheless, the pre-trained model can learn features that are more discriminative (for this task) than those pre-trained on the different dataset of ImageNet. This transfer learning behavior again resembles those seen semantic classification tasks.
case
accuracy
case
transfer acc
supervied
82.9
random weights
6.7
linear probing w/ MAE trained on IN-IK
Y+C+D
27.7
76.2
Y+C+D+W
34.2
MAE trained on YCD
78.4
Y+C+D+W+L
34.2
Y+C+D+W+L+I
34.8
Table Self-supervised pre-training, fol- lowed by linear probing, achieves high ac- curacy for dataset classification. Here, we study MAE (He et al., 2022) as self- supervised pre-training baseline, which uses ViT-B as the backbone. The fully-supervised baseline for dataset classification with the same ViT-B architecture (82.9%). Results are on the YCD combination.
MAE (Hl et al., 2022)
68.0
MoCo v3 (Chen et al., 2021)
76.7
Table 8: Features learned by classifying datasets can achieve nontrivial results un- der the linear probing protocol. Transfer learning (linear probing) accuracy is reported on ImageNet- using ViT-B as the backbone in all entries. The acronyms follow Table

---

Published as a conference paper at ICLR 2025
5.4 FEATURES LEARNED BY CLASSIFYING DATASETS
We have shown that models trained for dataset classification can well generalize to unseen validation data. Next we study how well these models can be transferred semantic classification tasks. To this end, we now consider dataset classification as pretext task, and perform linear probing on the frozen features on semantic classification task (ImageNet- 1K classification). Table shows the results of our dataset classification models pre-trained using different combinations of datasets."
Comparing with the baseline of using random weights, the dataset classification models can achieve non-trivial ImageNet- IK linear probing accuracy. Importantly, using combination of more datasets can increase the linear probing accuracy, suggesting that better features are learned by discovering the datasel biases across more datasets.
As reference, it should noted the features learned by dataset classification are significantly worse than those learned by self supervised learning methods, such as MAE (He et al., 2022) and MoCo v3 (Chen et al., 2021), which as expected. Nevertheless, our experiments reveal that the dataset bias discovered by neural networks is relevant to semantic featurrs that are useful for image classification.
5.5 CROSS-DATASET GENERALIZATION
Torralba & Efros (2011) observed that models often struggle to generalize across different datasets. For instance, model trained on dataset A recognize cars may perform well on held-out images from dataset A but poorly on dataset B. We revisit this cross roos-generaalizttonn experiment using the modern and large-scale datasets we study. Due to the lack of a common task defined on them, we use contrastive learning (MoCo v3) (Chen et al., 2021) as a surrogate task and report their validation losses. More result with masked autoencoding (MAE) (He et al., 2022) is available in Appendix C.
train / oval
YFCC
CC
DataComp
WTT
LAION
ImageNet
average
YFCC 1.761
2.202
2.668
2.083
2.764
2.026
2,251
CC
1.971
1.759
1.885
2.012
1.874
1.970
1.912
DataComp
2.216
1.891
1.772
2.161
1.801
2.023
1.977
WTT
1.969
2.059
2.238
1.742
2.288
2.004
2.050
LAION
2.332
1.902
1.787
2.236
1.779
2.097
2.022
ImageNet 1.941
2.077
2,040
2.157
2.233
1.742
2.032
combined
1.940
1.841
1.822
1.915
1.847
1.860
1.871
Table 9: Cross-dataset generalization with MoCo v3 validation losses. Bold indicates the lowest for each evaluation dataset (column). A clear diagonal indicates that cross-dataset transfer always has a gap with training on the same dataset. Combining all datasets yields the best result when averaged.
Table 9 shows the results. We can see clear diagonal with the lowest validation loss in each column. This indicates that on any particular validation dataset, only pre-training the same training dataset can achieve the best generalization performance. Therefore, despite larger and more diversified datasets, cross-dataset generalization remains problem. Interestingly, simply combining all datasets (controlling the total number of images by taking 1/6 images from each) yields the best overall result. This suggests combining datasets may be a simple strategy to reduce dataset bias.
CONCLUSION
We revisit the dataset classification problem in the context of modern neural networks and large-scale datasets. We observe that the datasets bias can still be casily captured by modern neural networks. This phenomenon is robust across models, dataset combinations, and many other settings.
It is worth pointing out that the concrete forms of the bias captured by neural networks remain largely unclear. We have discovered that such bias may contain some generalizable and transferrable patterns, and that it may not be easily noticed by human beings. We hope further effort will be devoted to this problem, which would also help build datasets with less bias in the future.
In this comparison, search for the optimal learning rate, training epoch, and the layer from which the feature is extracted, following the common practice in the self-supervised learning community.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. We thank Yida Yin, Mingjie Sun, Saining Xie, Xinlei Chen, and Mike Rabbat for valuable discussions and feedback, and all volunteers for participating in our user study.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arxiv preprint arXiv:1607.06455, 2016.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, poroggrppyy, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 2018.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised Vision Transformers. In ICCV, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. In NeurIPS Datasets and Benchmarks Track, 2021.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
Logan Engstrom, Andrew Ilyas. Shibani Santurkar, Dimitris Tsipras. Jacob Steinhardt, and Alek- sander Madry. Identifying statistical bias in dataset replication. In ICML, 2020.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. IJCV, 2010.
Li Fei-Feii, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR workshops, 2004.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. In NeurIPS Datasets and Benchmarks Track, 2023.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLRR 2016.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
11

---

Published as a conference paper at ICLR 2025
Google People Al Research. Know your data. https:/ htt.://////.oooooraattttttoooooo 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2018.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Andrej Karpathy, What learned from competing against a ConvNet on Ima- geNet. batpss//karpatllyggthbbb..e http://////////att---aeeaaaeeee/ /wwht---leaaeeddffffff competing-agaiinss --ooavvoo--------ggeeeeee... 2014.
Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster. Reduced, reused and recycled: The life of a dataset machine learning research. NeurIPS, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In ICML, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi- multi-domain semantic segmentation. In CVPR, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied document recognition. Proceedings of the IEEE, 1998.
Tsung-Y. Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, 2022.
David G Lowe. Distinctive image features from scale-invariant keypoints. IJC, 2004.
Shira Mitchell, Eric Potash, Solon Barocas, Alexander D' A'mour, and Kristian Lum. Algorithmic fairness: Choices, assumptions, and definitions. Annual review of statistics and its application, 2021.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. NeurIPS, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics Image Processing, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. CVPR, 2012.
12

---

Published as a conference paper at ICLR 2025
Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision? In WACV, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Ai and the everything in the whole wide world benchmark. arXiv preprint arX1v:2111.15366, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Lawrence Gilman Roberts. Machine Perception of Phree-Dimensional Solids. PhD thesis, Mas- sachusetts Institute of Technology, 1963.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 2015.
William F Schreiber. Image processing for quality improvement. Proceedings of the IEEE, 1978.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS Datasets and Benchmarks Track, 2022.
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv:1711.08556 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
Yu Sun, Xiaolong Wang. Liu Zhuang, John Miller, Moritz Hardt, and Alexci A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data multimedia research. Communications of the ACM, 2016.
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look dataset bias. arxiv preprint 1505v 1505.0/257, 2015.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. ICML, 2021.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint 14Xiv: 2014.
13

---

Published as a conference paper at ICLR 2025
CWJ van Miltenburg. Stereotyping and bias in the flickr30k dataset. In Workshop maltimodal corpora: computer vision and language processing. 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Andrea Vedaldi and Andrew Zisserman. Efficient additive kernels via explicit feature maps. IEEE transactions on pattern analysis and machine intelligence, 2012.
Hermann Von Helmholtz. Optique physiologique. Masson, 1867.
Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky. REVISE: A tool for measuring and mitigating bias in visual datasets. IJC,, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness visual recognition: Effective strategies for bias mitigation. In CVPR, 2020.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Conference on fairness, accountability and transparency, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurlPS, 2014.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In JCCV, 2019.
Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernandez Dominguez. Wilddash-creating hazard-aware benchmarks. In ECCV, 2018
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AAAI/ACM Conference on AI, Ethics, and Society, 2018 8a.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018b.
Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. In ICCV, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 2017.
14

---

Published as a conference paper at ICLR 2025
A
USER STUDY
To have a better sense of the dataset classification task, we further conduct a user study to assess how well humans can do this task and to learn their experience.
Settings. We ask our users to classify individual images sampled from the YCD combination. Because users may not be familiar with these datasets, we provide interface for them to unlimitedly browse the training images (with ground-truth labels of their dataset identities) when they attempt to predict every validation image. We ask each user to classify 100 validation images, which do not overlap with the training set provided them. We do not limit the time allowed to be spent on each image or on the entire test.
Users. group of 20 volunteer participants participated in user study. All of them researchers with machine learning background, among which 14 have computer vision research experience.
User study results. Figure 5 shows the statisties of the user study results the dataset classification task. In summary, 11 out of all 20 users have 40%-45% accuracy. 7 users have 45%-50%, and only 2 users achieve over 50%. The mean is 45.4% and the median 44%.
The human performance is higher than the chance-level guess (33.3%), suggesting that there exist patterns that humans can discover to distinguish these datasets. However, the human performance is much lower than the neural network's ' 84.7%.
We also report that the 14 users who have com- puter vision research experience average per- form no better than the other users. Among these 14 users, we also ask the question "What accuracy do you expect a neural network can achieve for this task?" The estimations are 60% from users, and 80% from users, and 90% from user; there were 5 users who chose not to answer. The users made these estimations before becoming aware of our work.
10
counts
35
40
50
60
accuracy (%)
Figure 5: User study results on humans per- forming the dataset classification task. Humans generally categorize images from YCD with 40- 60% accuracy.
There are 15 participants who describe the difficulty of the task as "difficult". No participant describes the task "easy". 2 participants commented that they found the task "interesting".
We further asked the users what dataset-specific patterns they have used to solve this task. We summarize their responses below, in which brackets indicate how many users mentioned each pattern:
YFCC: people (6), scenery (3), natural lighting, plants, lifestyle (2), real-world, sport, wedding,
high resolution (2), darker, most specific, most new, cluttered;
CC: cartoon (2), animated, clothing sample, product, logo, concept, explanatory texts, geography, furniture, animals, low resolution, colorful, brighter, daily images, local images, single person, realistic, clean background;
DataComp: white background (3), white space, transparent background, cleaner background, single item (2), product (2), merchandise, logo-style, product showcase, text (2), lots of words, artistic words, ads, stickers, animated pictures (2), screenshots, close-up shot, single person, people, non non-realistic icons, cartoon, retro.
In these user responses, there are some simple types of bias that can be exploited (e.g., "white background" for DataComp), which can help increase the user prediction accuracy over chance- level guess. However, many types of the bias, such as the inclusion of "people" in images, are not meaningful for identifying the images (e.g., all datasets contain images with people presented).
B IMPLEMENTATION DETAILS
For image-text datasets (CC, DataComp, WIT, LAION), we only use their images. The LAION dataset filtered before usage. We uniformly sample the same number of images from each dataset
15

---

Published as a conference paper at ICLR 2025
to form the training/ validation sets for dataset classification. If a dataset already has pre-defined training/ validation splits, we only sample from its train split. 1M images for each dataset is used as the default unless otherwise specified. This is not small collection, yet it still only represents a tiny portion of images (e.g.. <10%) for most datasets we study. To speed up image loading, the shorter side of each image is resized 500 pixels if the original shorter side is larger than this. We observe this has minimal effect on the performance of models.
We train the models for the same number of samples seen as in typical 300-epoch supervised training on ImageNet-IK classification (Liu et al., 2022), regardless of the number of training images. This corresponds to the same number of iterations as Liu (2022) since the same batch size used. The complete training recipe is shown in Table 10.
config optimizer
value
AdamW
learning rate
le-3
weight decay
0.3
optimizer momentum
B1, 3.9. 0.95
batch size
4096
learning rate schedule
cosine decay (ImageNet-IK
warmup epochs
training epochs
300 (ImageNet-1K)
randomaug (Cubuk et al., 2020)
(9, 0.5)
label smoothing
0.1
mixup (Zhang et al., 2018b)
0.8
cutmix (Yun et al., 2019)
1.0
Table 10: Training settings for dataset classification.
For the linear probing experiments on ViT-B in Section 5.3 and 5.4, we follow the settings used in the checkpoint from epoch 250, and sweep for base learning rate from (0.1, 0.2, 0.3), and a layer index for extracting features from (8, 9, 10). For the cross-dataset generalization experiment Section 5.5, we use batch size of 1024 and keep all other hyper-parameters the same as those in MoCo v3 (Chen et al., 2021)
MAE (He et al., 2022). For Section 5.4,
During inference, an image first resized so that its shortest side is 256 pixels, maintaining the aspect ratio. Then the model takes its 224 224 center crop input. Therefore, the model cannot directly exploit the different distributions of resolutions and/or aspect ratios for different datasets as a shortcut for predicting images' dataset identities. The model takes randomly augmented crops of 224x224 images as inputs training.
C ADDITIONAL RESULTS
Training Curves. We plot the training loss and validation accuracy for ConvNeXi-T YCD classifica- tion Figure 6. The training converges quickly to a high accuracy level in the initial phases. This again demonstrates neural networks' strong capability in capturing dataset bias.
val accuracy
80
val accuracy
60
train loss
0.9
train
40
0.7
0
20K
40K
60K
80K
training iterations
Figure 6: Training curves for YCD classification. The model converges quickly.
ImageNet vs. ImageNetV2. ImageNetV2 (Recht et al., 2019) attempts to create a new validation set trying to follow the exact collection process of ImageNet-I IK's validation set. As such, the images look very much alike. We find a classifier could reach 81.8% accuracy classifying ImageNetV2 and ImageNet- 1K's validation training from each. This again demonstrates how powerful neural networks are at telling differences between seemingly close image distributions.
substantially higher than 50%, despite only using 8K images for
16

---

Published as a conference paper at ICLR 2025
ground-truth WIT DataComp CC YFCC
80.7%
4.0%
0.5%
12.5%
0.4%
1.9%
4.3%
80.6%
3.4%
4.1%
5.8%
1.7%
1.8%
17.0%
44.1%
2.3%
29.2%
5.7%
13.1%
3.7%
0.9%
77.9%
0.9%
3.5%
LAION
0.9%
15.6%
23.8%
1.9%
64.4%
3.5%
DataC.
1.7%
2.2%
5.1%
2.7%
2.0%
86 .2%
YFCC
CC
DataComp WIT prediction
LAION ImageNet
Figure 7: Confusion matrix for the 6-way classification in Table 2.
Confusion Matrix. We plot the confusion matrix for the 6-way dataset classification. We observe there exists a high confusion between DataComp and LAION. This is likely because they (Schuhmann et al., 2022; Gadre et al., 2023) both source from Common Crawl and apply filtering to select images that align closely with their captions in the CLIP (Radford al., 2021) embedding space.
Cross-dataset Generalization with MAE. In addition contrastive learning (MoCo v3) task used in Section 5.5, here we explore dross-dttaset generalization problem with another surrogate task, masked autoencoding (MAF) (He al., 2022). Table 11 shows the results. Similar to our previous observations, there is clear diagonal pattern with the lowest validation loss in each column, although the differences in loss values in each column are much smaller than that in contrastive learning. This indicates that the model pre-tre re-irrd given dataset only generalizes well within the same dataset but less well when transferred to others.
train / eval YFCC
YFCC
CC
DataComp
WIT
LAION
ImageNet
average
0.419
0.394
0.320
0.434
0.332
0.397
0.383
CC
0.423
0,386
0.311
0.433
0.320
0.395
0.378
DataComp
0,428
0.393
0.306
0.437
0.317
0.394
0.379
WIT
0.423
0.394
0.317
0.427
0.328
0.396
0.381
LAION
0.429
0.392
0.306
0.439
0.314
0.395
0.379
ImageNet 0.425
0.395
0.312
0.437
0.325
0.389
0.380
combined
0.422
0.388
0.306
0.430
0.317
0.391
0.376
Table 11: Cross-dataset generalization with MAE validation losses. Bold indicates the lowest for each evaluation dataset (column).
D LIMITATIONS
While we have discovered that modern neural networks can achieve an excellent accuracy in classify which dataset an image is from, is still unclear what are the exact forms of bias captured by neural networks. This warrants future research on understanding and interpreting the concrete forms of bias. addition, this work examines limited many other popular image datasets, as well as datasets on other domains like videos and language.
six large-scale image datasets, while leaving out
17


Reviews:
[{"review_id": "UxJyae4G5K", "reviewer": "Reviewer_Bad9", "summary": "This paper presents a modern version of the dataset classification experiment proposed by [Torralba & Efros (2011)](https://people.csail.mit.edu/torralba/publications/datasets_cvpr11.pdf): training a neural network to assign images to the right dataset it comes from. SVM reached reasonable accuracies by then on image classification datasets -- indicating dataset bias exist.\n\nThis paper experiments with modern backbones (eg, ConvNeXt and ViT) on modern large-scale datasets (eg, WIT, LAION), finding that dataset classification becomes even easier -- dataset bias is easier to spot by models despite the datasets being diverse and large-scale. The authors tried to explain this phenomenon, but it turns out the bias is not attributed to low-level clues and models do not simply memorize the data -- they learn high-level signals. The authors also show that self-supervised representations -- which tend to be good in semantics -- also work well on dataset classification, indicating that the dataset bias can be attributed to some high-level pattern.", "questions": "- To me there might never be a simple answer to the question of what kind of signal models learn when they succeed in discriminating the dataset one image comes from. Modern neural networks are so good at overfitting and any subtle patterns in dataset curation -- no matter low-level or high level, could be captured by them and processed to a strong black-box representation. If we accept that there is bias between CC, WIT, and LAION, and the bias remains as these datasets grow in scale, then what kind of harm would there be? If the representations learned from them generalize well, is the bias no longer a concern?\n\n- When we only consider datasets at a very large scale (eg, >100M), will it still be easy for models to tell the difference?\n\n- Dataset classification may not be a universal indicator of the bias between datasets, as it only sees one sample each time. Other aspects like concept frequency distribution (motivation of MetaCLIP) are also an important aspect of dataset bias and is highly related to the quality of representations learned upon. Do we need some new surrogate task/indicator for measuring bias in modern datasets in the next decade?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper presents a timely revisiting of an old question: how biased are ML datasets in the view of ML models? While it is widely accepted that some modern datasets are substantially diverse and large-scale, they can still be easily discriminated by modern models. This is a fresh conclusion and could trigger meaningful rethinking about the bias of large-scale datasets and models trained upon.\n\n- The study is comprehensive and well-presented. It covers most modern backbones and pertaining datasets, and studies various factors that affect dataset classification performance. It is also accompanied by a thorough analysis that considers what kind of clue networks are used to discriminate these datasets, what kind of representation is beneficial, and how well models generalize across datasets, etc.", "weaknesses": "The way this paper is presented might be over-focused on the dataset classification task itself, and it might require readers to read [Torralba & Efros (2011)](https://people.csail.mit.edu/torralba/publications/datasets_cvpr11.pdf) first to gain a full picture. In the context of that paper, the task was proposed to show that image classification datasets by then are biased and cannot make a complete representation of the visual world. They discussed possible sources of such bias (eg, in data selection), but more importantly it also led to discussions on how to measure the value of a dataset (cross-dataset generalization), how such biases shape representations (eg, different negative sets make models capture different aspects of representations), and how to overcome such biases (eg, random cropping augmentation). If without considerable discussions on the latter, it might be unclear for readers to understand the significance of studying the dataset classification task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6QE4oqrBf", "reviewer": "Reviewer_mF8E", "summary": "The paper studies the phenomenon of dataset bias through the lens of dataset classification. In this work, the authors expand upon the \"Name the dataset\" experiment from Torralba and Efros (2011) to modern neural networks. They then observe that modern NNs are relatively successful at the task of dataset classification, that is, predicting the source dataset of an image, under a variety of settings. The claim is that dataset bias still exists as NNs can exploit minor differences between large-scale image datasets to successfully solve the problem. The authors further test upon their hypothesis by designing and evaluating ConvNext, ViTs, and other models under a variety of confounding factors.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well written, and clearly conveys the experiments as well as the intuition behind each one. \n\n2. The experiments are well-designed and clearly support the conclusions. While the underlying idea of `dataset classification' is not new, the authors have extended the experiments to clearly account for a variety of confounding factors.\n\n3. It is also interesting to observe that the dataset bias task is also learning semantically useful features similar to pretraining, and self-supervised learning objectives. This is of independent interest as it can be used as a quick, cheaper method to initialize networks. While performance lags behind other approaches, this opens up some interesting ideas towards using dataset bias as an exploiting factor for self-supervision in future work.\n\n4. The model generalization vs memorization experiments show a significant vulnerability in deep networks exploiting dataset-specific features, with methods like augmentation failing to correct for the same.", "weaknesses": "Overall, I found the paper to be well-written and clear. However, I am listing some points for improvement,\n\n1.  It might be a useful exercise to use unbalanced mixes of the datasets as well, to observe effects of bias when a model encounters sources with varying numbers of images.\n\n2. I also suggest adding confusion matrices for the experiments in Table 2 to help understand correlations between datasets.\n\n3. The paper also does not address any directions on how to specifically tackle dataset bias. Given that standard approaches like data augmentation do not work, I am curious to know the authors' thoughts on methods like synthetic data generation, or domain generalization approaches as possible solutions to the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D833w4nlBx", "reviewer": "Reviewer_uFB6", "summary": "The paper revisits the \"dataset classification\" experiment from Torralba and Efros 2011 and applies it to modern neural networks and web-scale datasets. The authors find out that modern deep networks are very capable of solving such an artificial task, with a recognition accuracy of up to ~90%.", "questions": "- Can a qualitative analysis be provided to explain why specific dataset classification tasks are significantly more challenging than others?\nFor instance, the ConvNeXt-T model on YFCC-CC-ImageNet scores 92.7%, whereas on CC-DataComp-LAION, it only reaches 62.8% (Table 2, top panel). Is this large discrepancy solely due to differences in distribution overlap? How could this phenomenon be visualized or understood more intuitively?\n\n- It would also be interesting to visualize the difference in features learned by the network via, e.g., intra-class visualizations in latent space [1] or CAM approaches, compared to features learned when performing an actual classification task.\n\n[1] Donglai Wei, Bolei Zhou, Antonio Torrabla, and William Freeman; Understanding Intra-Class Knowledge Inside CNN\n\nOverall, the paper's strengths, including the finding of high dataset classification accuracy at large scales and the extensive, well-executed experiments, outweigh its weaknesses.\nThe findings interest the community, and the work is clear and well-presented. While addressing the identified weaknesses, such as providing more qualitative insights into the learned features and biases, which would strengthen the contribution, I am leaning towards acceptance. I would be inclined to raise the score if these aspects are improved.\n\n---POST REBUTTAL---\n\nThe authors provided an extensive rebuttal covering my main concerns. For this reason, I am updating the score. I encourage to include the additional analyses performed in the final manuscript to strengthen the contribution.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The paper's main finding, i.e., that the accuracy for the dataset classification task can be as high as 90% is surprising considering the size and diversity of the experimented web-scale datasets.\n\n2) The authors perform a comprehensive set of experiments to cover multiple settings and ensure the validity of their findings, including 20 dataset combinations, several model sizes, architecture types, data augmentation strengths, and the number of available data points.\n\n3) In Section 5, the paper provides additional analyses that rule out the possibility of achieving such high accuracy due to low-level signatures. Through a pseudo-dataset experiment and self-supervised learning, the authors also demonstrate that the networks are indeed learning dataset-related \"semantic\" features, further supporting their findings.\n\n4) Overall, I found the paper well-written, clear, and easy to follow.", "weaknesses": "1) The paper is repeating an experiment that was introduced roughly 10 years ago but at a larger scale. Hence, despite the surprising practical result, the paper does not introduce a novel concept in absolute terms, and it does not propose a method to mitigate/decrease dataset bias (only observations and not practical solutions). \n\n2) The paper lacks (even controlled) experiments to offer a qualitative evaluation of potential biases present in large-scale modern datasets. Looking at Table 2 (top panel), the classification performance significantly varies up to 30 percentage points.\n\n3) Although the authors repeatedly assert, with empirical backing, that the networks can learn \"semantic\" features for dataset classification, they do not characterize these features in practice, even qualitatively. This absence of practical insight leaves a gap in understanding how the networks perform these tasks.\n\nSee Questions for possible improvements and suggestions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)