Understanding Bias in Large-Scale Visual Datasets
Authors: Boya Zeng
Yida Yin
Zhuang Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: dataset bias
visual datasets
Primary area: machine_vision

Abstract: A recent study has shown that large-scale visual datasets are very biased: they can be easily classified by modern neural networks. However, the concrete forms of bias among these datasets remain unclear. In this study, we propose a framework to identify the unique visual attributes distinguishing these datasets. Our approach applies various transformations to extract semantic, structural, boundary, color, and frequency information from datasets, and assess how much each type of information reflects their bias. We further decompose their semantic bias with object-level analysis, and leverage natural language methods to generate detailed, open-ended descriptions of each dataset's characteristics. Our work aims to help researchers understand the bias in existing large-scale pre-training datasets, and build more diverse and representative ones in the future. Our project page and code are available at boyazeng.github.io/understand_bias.

Understanding Bias in Large-Scale Visual Datasets
Boya Zeng" University of Pennsylvania
Yida Yin UC Berkeley
Zhuang Liu Meta FAIR
"equal contribution
Abstract
A recent study [40] has shown that large-scale visual datasets are very biased: they can be easily classified by modern neural networks. However, the concrete forms of bias among these datasets remain unclear. In this study. we propose framework to identify the unique visual attributes distinguishing these datasets. Our approach applies various transformations extract semantic, structural, boundary, color, and frequency information from datasets, and assess how much each type of information reflects their bias. We further decompose their semantic bias with object-level analysis, and leverage natural language methods to generate detailed, open-ended descriptions of each dataset's characteristics. Our work aims to help researchers understand the bias in existing large-scale pre-training datasets, and build more diverse and representative ones the future. Our project page and code bias.
are available at boyazeng, www.gb..i/.hdeeeee.
1 Introduction
Recently, Liu and He [40] revisited the "Name That Dataser" experiment introduced by Torralba and Efros [69] in 2011, which highlighted the built-in bias of visual datasets. It is classification task where each dataset forms a class, and models are trained to predict the dataset origin of each image. The datasets back in 2011 were found to be classified quite accurately by SVMs [69]. Since then, significant efforts have been devoted to creating more diverse, large-scale, and comprehensive datasets [15, 37, 35, 60]. Surprisingly, a decade later, the largest and supposedly most diverse datasets (e.g., YFCC [66], CC [1 1J. DataComp [19]) can still be classified with remarkably high accuracy by modern neural networks [40].
Although we now know these large-scale datasets are very biased, a lingering question remains: what the concrete forms of bias among them', that cause them to be easily classified? Understanding the bias among datasets is essential for addressing it, and for improving dataset diversity and coverage. Creating datasets that more comprehensively represent the real world is challenging yet crucial (64, 55, 31 onnly then can we build truly general-purpose vision systems-systene capable of handling various scenarios out of the box, and performing reliably in real-world situations [30, 29, 24].
To this end, we develop a framework for understanding the concrete forms of bias among datasets. We isolate the semantic, structure, boundary, color, and frequency information through various transformations. For example, transforming an image into a semantic segmentation map preserves semantics while discarding most texture information. We then perform the dataset classification task on the transformed datasets, to quantify how each type of information reflects the dataset bias.
To pinpoint the semantic bias in datasets, we further conduct object-level and open-ended language analysis. Specifically, we leverage pre-trained recognition models to identify objects that characterize each dataset. In addition, using a Vision-Languaae Model (VLM), we generate image captions as
Note that in this work study bias among datasets. This concerns the coverage of concepts and content (i.e., how representative the dataset is for the real world). This is related to, but different from, another notion of bias that concerns social stereotypical bias, as well as algorithm faimness [8, 75, 6, 47].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

surrogate language representations of the images. We then apply topic models and Large Language Models (LLMs) to generate natural language descriptions for each dataset.
We apply our framework to three popular large-scale visual datasets: YFCC, CC, and DataComp, following [40]. Surprisingly, after transforming images into various semantic and structure represen- tations (e.g.. object bounding boxes and contours), neural networks can almost always still accurately predict their dataset origin. This highlights the bias in semanties and object shapes. Our object-level queries further reveal discrepancy in object diversity and distribution across the YCD datasets. Lastly, open-ended language analysis indicates that YFCC emphasizes outdoor and natural scenes with human interactions, while DataComp features digital graphics heavily.
Our framework operates images only and does not require any human annotations, making it compatible with any image dataset. It can be applied in future dataset curation to assess data diversity, and guide the inclusion of data with various attributes. We hope this work can help researchers address dataset bias based on their needs, and develop more inclusive and diverse visual datasets.
2 Related Work
Dataset classification. The dataset classification problem was originally proposed by Torralba and Efros [69] in 2011. Tommasi al. [68] later also studied this problem with linear classifiers using pre-trained CNN features. In contrast to prior work focusing on labeled smaller-scale datasets with shared object classes, Liu and He [40] recently revisited the dataset classification problem with large-scale, diverse, and presumbly less-biased datasets. They demonstrated that modern neural networks are still excellent at capturing bias among these datasets. Our work identifies the exact forms of the bias among datasets beyond dataset classification results.
Social bias and fairness. Extensive literature has highlighted that various visual datasets underrep- resent certain demographic groups [70, 75, 51], contain various gender stereotypes [81, 28, 45], or ignore some geographical regions [62, 75, 71]. These social fairness issues in datasets result in the deployment of flawed models [8, 72, 6] that may produce biased predictions or struggle to generalize well across different domains. Instead of focusing on social fairness in each dataset, we study the representativeness (i.e., coverage of real-world concepts and objects) and understand its differences among datasets. Note that Meister el al. [45] also use transformations to isolate different types of information, with specific focus on gender bias in datasets.
Bias detectors and debiasing tools. There are approaches that can locate the imbalance of object representation within datasets [23, 71]. Dataset rebalancing methods [8, 75] seek to correct these representation imbalances across protected attributes. Algorithmic intervention and regularization, such as adversarial training [78, 79, 44] and domain- inmaiinnnddeeddeent training [73], can counteract the propagation of bias and stereotypes in downstream modeling. Note that these prior methods often require ground-truth annotations to identify or mitigate potential bias, while our framework can analyze unlabeled pre-training datasets and provide insights beyond object distribution imbalances.
3 Isolating Bias with Transformations
Although modern neural networks can achieve excellent accuracy in the dataset classification prob- lem [40], what bias is captured by neural networks remains unclear. To better understand this, selectively preserve or suppress specific types of information using various transformations. We then train a new model on the transformed datasets to perform the dataset classification task. As result, its dataset classification performance indicates the level of bias in the extracted information. For example, transforming an image into depth map captures the spatial geometry while discarding texture, allowing us to assess bias present solely in spatial information.
3.1 Datasets and Settings
Based on Liu and He [40], we take YFCC100M [66]. CC12M [11], and DataComp- IB [19] (collec- tively referred to as "YCD") and study their bias in this work. Figure 1 shows example images.
Our training setup also adapted from [40]. Specifically, we randomly sample 1M and 10K images from each dataset as training and validation sets, respectively. We employ the same ConvNeXt-Ting
2

---

YFCC
CC
DataComp
82.0%
original
20.3%
Figure Original images. We sample two images from each of YFCC [66], CC [1I], and DataComp [19]. Dataset classification on the original images has reference accuracy of 82.0%.
image classification model [41] and train it for 30 epochs to classify the combined dataset with 3M samples. Note that the original work [40] achieves 84.7% accuracy on the validation set, while we achieve 82.0% accuracy due shorter training (roughly 23% of original length). We refer to this 82.0% as the reference accuracy in this paper. For all image transformations, we use the same ConyNeXt- model and almost identical recipes (details in Appendix A.I). We repeat data sampling and experiments three times, reporting mean validation accuracy and standard deviation.
3.2 Semantics
To start, we seek to understand how semantically biased the datasets are. Specifically, we extract semantic components from the images with fine-grained, coarse-grained, we apply a variational autoencoder, potentially reducing low-level signatures (e.g., color quantization, JPEG compression). Figure 2 shows the transformations and their dataset classification accuracies.
no spatial detail. Also,
YFCC
CC
DataComp
sem.
69.8%
seg.
1:066%
obj.
61.9%
det.
1:045
63.8%
A black and white dog
A herd of elephants walking through grassy field.
A room with bed, desk, chair, TV, and a vase with flowers.
A large wooden dining table with wicker chairs and a chandelier above it.
A notebook
food truck is parked on the side of the road.
1.01%
with polka (short)
image cap.
standing stream
dots and pink
and blue book 66.1%
water.
on table.
10.2%
(long)
VAE
77.4%
+0.3%
Figure 2: Transformations preserving semantic information (semantic segmentation, object detection, and caption) and potentially reducing low-level signatures (VAE) result in high dataset classification accuracy. This suggests that semantic discrepancy an important form of dataset bias.
Semantic segmentation offers fine-grate tne-grained semantic annotation with rich object information by assigning a class label to each pixel. We take semantic segmentation model ViT-Adapter- [12] (with BEiT-v2 [53] backbone) trained on ADE20K [83] with 150 semantic classes (e.g., wall, building, sky, etc.) to generate semantic segmentation mask for each image. This mask is represented as an RGB image using color palette for different classes, as shown Figure 2. The model trained on this color-coded mask achieves 69.8% accuracy, well above the chance level.

---

Object detection provides coarse spatial annotations for objects through rectangle bounding boxes. We use Vigeet-Huge [36] trained on LVIS [25] with 1203 object categories to derive bounding boxes from each image. We keep object class names on the bounding boxes to account for semantic meaning. This representation reaches 61.5 .1.% dataset classification accuracy, below semantic segmentation.
Image captioning discards all visual information and produces semantic representations through natural language. Captions are less affected by pixel variations and spatial cues in images. Using LLaVA 1.5 [39, 38], we generate a single-sentence caption and a long-paragrapi caption for each image. Short captions are shown in Figure 2, and long captions are Appendix D. We finetune sentence embedding model Sentence T5-base [50] to perform dataset classification on these captions. This results in 63.8% 33.8% accuracy on short captions and 66.1 on long ones. both nearing the accuracy for semantic segmentation. The richer details in longer captions enhance performance.
Variation autoencoder (VAE) [33] encodes each image into latent vector and then reconstructs the original image from it. The datasets may use different JPEG compressions or image resolutions, which could be exploited shortcuts by dataset classification models. However, VAE's low-dimensional latent space may encode semantic information and suppress such low-level signatures. Reconstructing the images with pre-trained VAE from Stable Diffusion [58] only slightly decreases the accuracy from 82.0% to 77.4 77.4%, suggesting that low-level bias may have smaller impact than semantic bias.
Semantic segmentation, object detection, and image captioning extract semantic information with decreasing levels spatial information. On the other hand, VAE could potentially reduce low-level signatures while preserving the semantics. The high accuracies of dataset classification models indicate that semantic bias is an important component of dataset bias in YCD.
3.3 Structures
Next, we analyze the dataset bias rooted in object shape and spatial geometry rather than object semantics. To capture such structural visual bias, we use the Canny edge detector and the Segment Anything Model (SAM) [34] to outline object contours, and the Depth-Anything- V2 model [76] to measure pixel-level depth. While contour delineates fine-grained object shape, and depth estima- tion offers relative object positions, both lack the rich object semantic details present in semantic segmentation masks and bounding boxes (e.g., object class). Figure visualizes the transformations.
YFCC
CC
DataComp
Canny edge
71.0%
102%
SAM
73.2%
(contour)
60.66
depth
73.1%
4.0.%%
Figure 3: Transformations outlining object shapes and estimating pixel depth. Dataset clas- sification achieves higher accuracies on object contours and depth images than semantic information, indicating that object shapes and spatial geometry vary significantly across YCD.
Canny edge detector [10] is a classical algorithm that outlines rough object boundaries by capturing sharp intensity changes. It removes noise with Gaussian filter, calculates intensity gradients, and applies non-maximum suppression form edges, represented binary mask. This results 71.0% classification accuracy, 11% below the reference accuracy (82.0%).

---

Segment Anything Model (SAM) [34] can provide high-qualit, object segmentation masks. We could then use them delineate cleaner and more accurate shapes of objects that are minimally affected by local pixel variations, compared to the Canny edge detector. Specifically, we use SAM with the Vir-Larre backbone to generate class-agnostic segmentation masks and identify boundaries by finding pixels whose surrounding pixels are not all from the same object. The classification accuracy on SAM contours (73.2%) is slightly higher than that on Canny edge (71.0%).
Depth estimation captures the scene's ' spatial geometry, offering fine-grained spatial context and relative object positioning. Like contours, it excludes explicit semantic information about objects. The Depth Anythin- V2 (ViT-L) model [76] generates pixel-level depth estimation, encoded as normalized grayscale image. The resulting 73.1% accuracy comparable that of SAM contours.
The dataset classification accuracies on object contours and depth are even higher than the ones on semantics. This shows that object shape and spatial geometry variations significant among YCD.
3.4 Spatial Permutations
To further understand the level of bias captured in spatial information as opposed to semantics, we keep the RGB values of all pixels in each image unchanged but shuffle the pixel positions disrupt spatial coherence. We shuffle each image on the pixel level and the patch level, following fixed order and random order for all images, Figure shows the images shuffled in random order.
YFCC
CC
DataComp
52.2%
20.8%
pixel shuf.
(rnd.)
58.5%
10.5%
(fixed)
80.1%
patch shuf.
10.2%
(rnd.)
81.2%
(16x16)
+0.3%
(fixed)
Figure 4: Transformations breaking spatial structure. Pixel shuffling drastically decreases dataset classification accuracy, but patch shuffling has minimal impact. This demonstrates that local structure important and sufficient for models to learn the patterns of each dataset.
Pixel shuffling obfuscates the image classifier with a per- mutation of the pixels and forces it to find patterns from the color distribution of pixels in each image. As expected, this significantly decreases the classification accuracy to 52.2% for the random order and 58.5% for the fixed order.
80
aecuaac 1 (%%
random
60
fixed
reference
Patch shuffling first divides each image into smaller patches and then rearranges the order of the patches. Con- sequently, it preserves more local spatial information. Here we vary the patch size and show the results in Fig- ure 5. The accuracies of the fixed order and the random order shuffling are almost identical when the patch size is larger than 1. Surprisingly, with a patch size of 16, we almost reach the 82% reference accuracy.
50
patch size
Figure 5: Effect of patch sizes. Dataset classification accuracy approaches the reference one with larger patch sizes.
The significant performance drop with pixel shuffling shows completely destructing the local structure in YCD can reduce its dataset bias to a large extent. However, the minimal accuracy decrease after shuffling patches of size 16 indicates patch-level local structures in spatial information is sufficient for identifying visual signatures of the YCD datasets.
3.5 RGB
The high classification accuracy after pixel shuffling implies discrepancy in pixel color distributions among YCD. To further assess this difference color statistics among datasets, we transform each image into its average value for each color channel. Figure 6 shows the resulting images.
5

---

YFCC
CC
DataComp
mean RGB
48.5%
2007%
Figure 6: Averaging each color channel. Even when the values of each channel in images are averaged, the model can still achieve non-trivial dataset classification performance.
12e-3-
YFCC
YFCG
YFCC CC
ground-rults
1%
11.1%
8.8%
denaty
R--1
DataComp
DateComm
DaxCamp
DanaC.CC
49.0%
18.5%
32.6%
19.1%
15.4%
5..5%
50
100
200
30
100
150
250
250
50
100
150
200
250
YFCC
CC prediction
DataComp
rod
green charnel Value
blue channel value
Figure 7: Distribution of mean RGB values and confusion matrix. YFCC's RGB values are overall smaller, while CC's and DataComp's 's are very similar. This also reflected in the confusion matrix of dataset classification mean RGB images, where YFCC can be classified very easily (indicated by the dark blue box on the top left), while there high confusion between CC and DataComp.
Mean RGB. We compute the mean RGB value for each image. This abstracts the pixel details into a constant RGB color map and forces the image classifier to only use color statistics. The model's accuracy on mean RGB images is 48.5%, about 15% higher than the chance-level accuracy of 33.3%.
In Figure 7, we visualize the distribution of mean RGB values for YCD. There only a moderate difference in mean RGB distribution between CC and DataComp. However, YFCC is much darker than CC and DataComp. This is further suggested by the confusion matrix of the dataset classification model trained on mean RGB images shown in Figure 7, where the model classifies YFCC accurately but shows more confusion when distinguishing between CC and DataComp.
3.6 Frequency
Neural networks tend exploit texture patterns when the recognition task is inherently about the semantics [32, 21, 20]. If we decompose visual signals by frequencies, high-frequency bands typically capture these texture patterns and sharp transitions, whereas low-frequency components represent general structure and smooth variations. To explore how different frequency components contribute dataset bias, we apply high-pass and low-pass filters the original images.
YFCC
CC
DataComp
low-
70.4%
pass
filter
+0.5%
high- pass filter
79.2%
10.0%
Figure 8: Transformations filtering to-reference accuracy. This indicates that dataset bias exists in different frequeeniis. The high-pass filtered images are equalized for better visualization.
and low-frequency components retain close-
High-pan filter and low-pass filter. To filter signals based on frequencies, we first perform a 2D Fast Fourier Transform on each grayscaled image to obtain its representation in the frequency domain. We then apply an ideal filter [22] with hard threshold radius of 40 in the frequency domain, so as to only keep either high (ie., high-pass filter) or low (i.e., low-pass filter) frequencies. The filtered results are finally inversely transformed the original grayscale domain, visualized in Figure Additional results and visualizations are in Appendix B.6.

---

The model trained on images with high frequencies kept has an accuracy of 79.2%. This is slightly better than the one trained on imgges with low frequencies kept, which has an accuracy of 70.4%. Both accuracies are close to the reference accuracy of 82.0%.
The high accuracy of models trained either frequency component indicates that visual bias in the YCD datasets exists in both low-frequency and high-frequency components.
3.7 Synthetic Images
Synthetic images hold significant potential in augmenting data for various vision tasks [27, 3, 67, 5, Diffusion models [56, 46, 58] can generate synthetic images. However, if dataset bias can be inherited from a diffusion model's training images to its generated images, bias may persist and even amplify in downstream tasks that use generated images for training. To assess the bias in synthetic images, we run dataset classification on images generated from diffusion models, illustrated in Figure 9.
YFCC
CC
DataComp
uncond. (custom- trained)
NIC EE - 108%
text- to- image (pre- trained)
58.1%
+0.4%
Figure 9: Synthetic images from unconditional and text-to-image generation models. Uncondi- tionally generated images can be classified with near-reference accuracy. Images from text-to-image diffusion models using short captions have reasonable dataset classification accuracy.
Unconditional generation. We train unconditional Diffusion Transformer (DiT) [52] on each individual dataset in YCD. The models learn to generate synthetic images from random noise. Dataset classification is performed on the combination of synthetic data generated from each model, resulting in very high accuracy of 77.6%, nearly matching the reference accuracy of 82.0%.
Text-to-Image generation on image captions potentially preserves the semantic bias in the original images. We generate synthetic images from the SDXL-Turbo [59] diffusion model, conditioned on short captions produced by LLaVA (Section 3.2). By converting the original images into caption text and then back to the visual domain, we only retain the semantics captured captions. Note that, unlike the unconditional generation experiment above, here we do not train own text-to-image model for each dataset; instead, we use the same pre-trained model for all datasets. We reach 58. 1 % accuracy, falling slightly short of 63.8% when directly classifying short captions.
The high classification accuracy from unconditionally generated images shows that synthetic images sampled from a diffusion model can inherit the bias in the model's 's training images. The ability to classify synthetic images generated by pre-trained text-conditional generation model further confirms that semantic discrepancy is major contributor to dataset bias.
Summary. We analyzed the impact of various transformations on dataset classification, identifying semantics and structures as important contributors to dataset bias. Patch-level local structure informa- tion is sufficient to classify YCD, with datasets differing even in color statistics, Bias spans across frequency components, particularly in high-frequency bands. Finally, we showed that bias can be inherited in synthetic images of diffusion models. More results are in Appendix B.
4 Explaining Semantic Bias among Datasets
In the preceding section, we explore transformations to extract various types of image information, each exhibiting varying levels of bias. Among them, semantic bias heavily contributes to the high accuracy in the dataset classification task [40]. In this section, we identify specific interpretable semantic patterns within each dataset through object-level and language- based analysis.
7

---

4.1 Object-level Queries
YFCC
DataComp
Figure 10:
Grad-CAM heatmap [82, 61] on the dataset classification model trained on original datasets. The model focuses on specific objects to determine the dataset origin of each image.
Grad-CAM [82, 61] highlights key regions in input image that explain neural network predictions. In Figure 10, applying Grad-CAM to the reference model (Section 3.1) shows that the model focuses on semantically meaningful objects: elephant herd in the third image, table and chair in the fourth image, and pen in the sixth image. This suggests that the model might have leveraged the object-level information recognize the dataset identity of each image. To better understand this, we apply models pre-trained on other vision datasets (ImageNet- K [15], LVIS [25], and ADE20K [83]) to provide object annotations for each YCD image, and analyze their object-level bias. As a result, the analysis below is in the context of 3 sets of object classes, defined by these 3 datasets.
Imbalanced object distribution. Imbalance in object distribution is common form of semantic bias. For each object class, we calculate the number of images in each dataset in YCD that contain the object class and their percentage share relative to all images with that object class. Note for LVIS and ADE20K models' output, we count each object class only once per image, even if multiple instances or pixels of the same object class are present. Figure 11 shows the top 8 object classes with the highest percentage of images from YFCC, CC, or DataComp. The dominance of a certain dataset within these classes highlights a considerable imbalance in object-level distribution across datasets.
YFCC
CC
DataComp
InaggerWee clesses
ADE26K clesses
Imageefet dasees
ADE20K linaggeell
ADE20K
hammerhead
water buttle
trafii light
gown
call
sushion
rotion
knob
sink
kile
ophane
eshiaan
quill
pillow
rug
swwllic
suitcasse
well
traffic light
streetightt
streeblight
agaw
monkey
armchair
face powder
TV set
flaor
unioycle
laryard
swning
lsmpsshdde
varse
surnscreen
cap
rug
nematode
backkack
stage
comics
bed
sota
binder
bolt
painting
parachutel
pole
notebook
armchaik
painting
grill
bed
stage
sweatsh
bus
heart
ma Ibag
drawer
milmrr
pole
polo shirt
biyycle
Jarsey
kase
bed
windsso
hhandle
food
0.6
0.6
0.5
0.5
0.5
0.3
0.3
0.3
proportion
Figure 11: Object classes with the highest proportions of YFCC, CC, or DataComp images. Less- feesufre classes are not shown. Most classes consist predominantly of images from one dataset.
Figure also shows that YFCC constitutes much higher proportions in its top object classes than CC and DataComp in their respective classes (note the different x-axis scales in each subplot). To further see this, we visualize the distribution of unique object class counts per image in Figure 12. The higher variety of objects in YFCC images shows notable gap in object diversity among YCD.
Interpretable dataset classification with objects. The coefficients of logistic regression model form a natural importance ranking of input features when the features are binary. To leverage this, we represent each image with a binary vector, where each element indicates the presence of specific object class from set of objects (i.e., ImageNet, LVIS, or ADE20K), We train logistic regression model to predict the dataset origin of the images based on their binary vector rep- resentations. This simple model achieves validation accuracies of 52.0% with ImageNet objects, 52.4% with LVIS objects, and 52.4% with ADE20K objects.
LVIS
ADE20K
30
unque object class
24
16
18
12
12
6
YFCC CC DataComp
YFCC
CC DataComp
Figure 12: Unique object classes per image. On average, YFCC contains the highest num- ber of unique objects in each image, followed by CC, while DataComp exhibits the lowest. Figure 13 shows the top objects based on logistic regression coefficients. It highlights outdoor infrastructures (e.g., traffic light, clock tower, telephone pole, and building) in YFCC and houschold
8

---

YFCC
CC
DataComp
imagaNerr olosses
LVIS classes
ADE20K cinases limageMat classes
LMI closses
ADEZOK classos
Image/iat classes
LVIS closses
ADE20K closses
hammarhead
stage person bag
gown camic book
lation face powter
microphoree streeliiht
monkey
botlle cap television
trade
light
studio couch quitt web sit
plaything cushion
mailbag switch bindar
house
unioycle
clock lower
dress
gnil
rug skyscrapar truck cusition
nematode
plasti bag
colling rock pola
ring
animal
vase biszer monitor
perachute
speciadiss
rubber erasee
horee
towel
pickk sunscreenn
slage
glass
jarsey
pillow
person
vault
telephone pole
puzzle
running
lular
pole
water bottle
bulletin
maiital
fairr slide
telephoree
plar
speaker
building
Lhhwwer curtain/
caming
sky
site
drawer
Figure 13: Object class ranking from logistic regression coefficients. The regression classifies images based on object presence. YFCC has more top objects related to outdoor scenes, while CC and DataComp focus on household items and products. Classes with low frequencies are not shown.
items, products, and digital graphics (e.g., doll, ring. vase, blazer, and website) in CC and DataComp. These rankings partially overlap with the list of objects that are much more prevalent in one dataset than the others (Figure 11). However, the object rankings also identify objects that are more balanced across datasets, since logistic regression receives more weight updates based on more common objects during training. Thus, it provides a complementary angle on object-level bias among the datasets.
4.2 Open-ended Language Analysis
The word clouds [48] in Figure 14 offer visual representation of the most prevalent phrases in the long paragraph captions of each dataset from Section 3.2. We observe several frequent phrases featuring human subjects (e.g., people. group, wearing) in YFCC, elements of indoor scenes (e.g.. room, and dining table) in CC, and focus on white background in DataComp. In this section, we will use the long captions proxy to dive deeper into the semantic themes in each dataset. DataComp
Unsupervised topic discovery. We treat each cap- tion as a document and apply the Latent Dirichlet Allocation (LDA) [7] for topic discovery to each dataset in YCD (with the number of topics set to 5). Figure 15 presents the top words for each topic. Notably, in YFCC, three topics (first, second, and fifth) feature words associated with outdoor scenes; in CC and DataComp, their topics cover "logo" and "design," suggesting the presence of digital graphics.
YFCC
CC
ma
n
Aurtoc us
pl aced set woman
thtte backka Keroune
Fea woman
man
assolayed
wear dain
wea ring
g group
llaced TApleS
room Trent.
Figure 14: Word clouds [48] on the 100 most frequent phrases in each dataset. Phrase size corresponds its frequency. DataComp
YFCC
CC
[building, scene, street, sign]
[roam, table, chairs, chair, dining] [design, background, colors, logo, display] dress] [woman, shirt, (scene, people, water, group, atmosphere] [scene, builing, car, person, dog)
|logo, background, design, book, colors] [scene, table, room, building, atmossheree] [car, scene, truck, background, kitchen] [background, table, design. box, bottle] [man, woman, scene, shirt, person]
[scene, field, game, person, dog)
[table, dining, [people, man, woman, scene, group) [scene, water, trees, tree]
items]
Figure 15: LDA-extracted topics for each generated caption set. Each row lists the top 5 words for topic. YFCC focuses on outdoor scenes, while CC and DataComp contain more digital graphics.
Large Language Model (LLM) summarization. To assess whether dataset bias in captions can be identified by LLMs with limited examples, we provide GPT-4o with 120 captions per dataset and ask it to infer the dataset origin of new captions. Inference is performed on one validation caption at time until accuracy stabilizes at 52.9% on 480 captions. Further details are in Appendix A.2.
This powerful ability of LLMS allows them to provide open-ended and detailed natural language explanations. Specifically, we procedurally prompt GPT-4o to extract dataset-specific characteristics from caption datasets and refine its answers into 5 bullet points, shown in Figure 16. In summary, YFCC is characterized by abundant outdoor, natural, and human-related scenes, while DataComp concentrates on static objects and digital graphics with clean backgrounds and minimal human presence. In contrast, CC blends elements of both YFCC's dynamic scenes and DataComp's static imagery. Appendix A.2 provides the entire prompt structure and the complete LLM summariza- tion output. We further verify the validity of the semantic features from LDA and LLM Appendix C.

---

YFCC
CC
DataComp
Group Dynamics and Activities This distribution frequently showcases groups people engaged activities such as playing music, attending events, parlicipating in sports, emphasizing social interactions and communal settings
1. Organized Indoor and Outdoor Scenes
Object-Focused Descriptions Captions prominently feature specific objects or products (e coffee mugs, toys, cars), often isolated against minimalistic backgrounds to highlight their characteristics.
Captions depict well-structured environments. including cazy bedrooms, dining areas, cityscapes, and architectural landmarks, emphasizing the arrangement and detail.
Urban and Social Settings Captions often describe dynamic environments filled with people and activity or public settings, such as busy city streets, transportation hubs, and social events.
2. Vibrant and Playful Visuals
Human Interactions and Social Events
Scenes frequently include vibrant, colorful, and playful elements, focuaing on visually appealing and lively imagery that captures attention.
Emphasis social and formal gatherings like weddings. concerts, and ceremonies, highlightlnn attire, decor, and lively atmosphere.
Serene Natural Settings
Close-Up and Detailed Views Descriptions often emphasize close-up shots, highlighting intricate details, textures, and designs of objects, with focus sesthetic and functional attributes.
Many images feature serene ouldoor environments, including natural landscapes, gardens, and bodies of water, highlighling calm peaceful atmosphere.
3. Vivid and Dynamic Elements Descriptins focus colorful lively scenes, with vibrant attire, festive settings, and engagements, emphasizing visual appeal and movement.
4. Detailed Environmental Context
4. Detailed Objects and Clothing
and Artistic Compositions
Emotions and Interactions
Creative and Artistic Themes
5. Simplistic and Isolated Backgrounds
Figure 16: LLM summarization of dataset features. The bullet points highlight outdoor, natural, and human scenes in YFCC and static objects and synthetic images in DataComp. CC contains both YFCC's dynamic scenes and DataComp's static images.
Summary. We leveraged closed-set object-level queries and open- language analysis to interpret the semantic bias among datasets. The object- based analysis identified objects indicative of each dataset within a predefined object set. On the other hand, natural language methods are able to provide open-ended explanations for the characteristics of each dataset with rich details.
5 Discussion
YCD have different sources: YFCC is selected with minimal filtering from user-uploaded images on Flickr.com, while CC and DataComp filter web-sourced images with high quality in caption, image, or their alignment. We build the interpretable semantic bias in Section 4 to discuss the dataset curation procedures, and provide potential suggestions.
Filtering based on reference dataset or model may inherit its bias. DataComp has the fewest unique objects per image (Figure 12). This is possibly because DataComp filters for images with visual content close to ImageNet data in the embedding space [19]. Therefore, the remaining images tend to be object-centric [4]. also filters for images that align well with its captions in CLIP [54] embedding space, therefore favoring certain types of images, e.g., images containing text. To mitigate this, we may use datasets or models that contain less bias themselves for filtering.
The source website's 's image collection mechanism can introduce bias. We note that YFCC is heavily skewed towards outdoor scenes and human interactions (Section 4.2). This bias likely stems from its reliance on single data source, Flickr.com, where user-uploaded content often focuses on personal photos, landscapes, and social interactions.
Web images naturally contain more digital graphics. Since CC and DataComp images are from Internet webpages, professionally created content like advertisements, infographics, and digital media are prioritized. Dataset users should evaluate if this composition aligns with the downstream goals.
6 Conclusion
We proposed framework to study the bias in large-scale visual datasets and used it to analyze three representative datasets. By classifying transformed images' dataset origin, we identified structures and semantics as key factors in dataset bias. We further investigated specific forms of semantic bias among datasets through fixed object queries, highlighting distinctive concepts characterizing each dataset. Lastly, we extracted key topics and natural language summaries for each dataset. We hope this framework and these findings can encourage further exploration of dataset bias and help improve diversity and representation in future datasets.
10

---

Acknowledgements. We would like to thank Kaiming He, Olga Russakovsky, Mingjie Sun, Kirill Vishniakov, and Zekai Wang for helpful discussions and feedback.
References
[1] Sina Alemohammad, Josue Casco-Rodrigune Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go MAD. In ICLR, 2024.
[2]
Anthropic. Claude 3.5 sonnet. https://ww. https://ww.atttrroi..cc com, 2024.
[3] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. TMLR,, 2023.
[4] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019.
[5] Quentin Bertrand, Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. ICLR, 2024.
[6] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misog- yny. pornography, and malignant stereotypes. arXiv preprint arXiv:21
2021.
[7] David M Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. JMLR, 2003.
[8] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, 2018.
[9] Stephen Butterworth. On the theory of filter amplifiers. Wireless Engineer, 1930.
[10] John Canny. A computational approach edge detection. IEEE Transactions on Pullern Analysis and Machine Intelligence, 1986.
[11] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, 2021. [12] Zhe Chen, Yuchen Duan, Wenhai Wang. Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023.
[13] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
[14] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR,
2005.
[15] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[16] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In NeurIPS, 2024.
[17] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[18) Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In CVPR, 2024.
[19] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. In NeurlPS Datasets and Benchmarks Track, 2023.
[20] Robert Geirhos, Mern-K Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature
Machine Intelligence, 2020.
[21] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
[22] Rafael C. Gonzales and Paul Wintz. Digital image processing (2nd ed.). Addison-Wesley
Longman Publishing Co., Inc., 1987.
[23] Google People + Al Research. Know your data, 2021.
[24] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
11

---

[25]
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[26]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.
[28]
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In ECCV, 2018.
[29]
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of duttof-distributionn generalization. In ICCV, 2021.
[30] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations.
ICI.R, 2019.
[31] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsua Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- lision-language representation learning with noisy text supervision. In ICML, 2021.
[32] Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv:1711.1 2017.
[33] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 14, 2022.
[34] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In ICCV, 2023.
[35]
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJC,, 2020.
[36] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
[37] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[38] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2023.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2024.
[40] Zhuang Liu and Kaiming He. A decade's battle on dataset bias: Are we there yet? arXiv preprint arxiv: 2403.0863,, 2024.
[41] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining
Xie. A convnet for the 2020s. In CVPR, 2022.
[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[43] David G Lowe. Distinctive image features from scale-invariant keypoints. I/CV, 2004.
[44] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In /CML, 2018.
[45] Nicole Meister, Dora Zhao, Angelina Wang, Vikram Ramaswamy, Ruth Fong, and Olga
Russakovsky. Gender artifacts in visual datasets. In ICCV, 2023.
[46] Midjourney. Midjourney.com. 2022.
[47] Shira Mitchell, Eric Potash, Solon Barocas, Alexander D' Amour, and Kristian Lum. Algorithmic
fairness: Choices, assumptions, and definitions. Annual review of statistics and its application, 2021.
[48]
Andreas Mueller. Wordeloud for python, 2023.
[49] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers, Mteb; Massive text embedding benchmark. In ACL, 2023.
[50] Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.0888 2021.
12

---

[51]
Joon Sung Park, Michael S. Bernstein, Robin N. Brewer, Ece Kamar, and Meredith Ringel Morris. Understanding the representation and representativennss of age in ai data sets. In Proceedings of the 202 AAAI/ACM Conference on AI, Ethics, and Society, 2021.
[52] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023,
[53] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEÏT v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arxiv: 2208.06366 2022.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[55]
Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Ai and the everything in the whole wide world benchmark. In NewrIPS, 2021.
[56]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022. [57] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese
BERT-networks. In EMNLP, 2019.
[58] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[59] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:231
2023.
[60] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS Datasets and Benchmarks Track, 2022.
[61] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
[62] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv: 108.60636, 2017.
[63] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurlPS, 2020.
[64] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. In ICCV, 2017.
[65] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions.
In CVPR, 2015.
[66] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 2016.
[67] Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. In CVPR, 2024.
[68] Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look at dataset bias. Domain adaptation in computer vision applications, 2017.
[69] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011.
[70] Emiel Van Miltenburg. Stereotyping and bias in the flickr30k dataset. arXiv preprint
1005066800 2016.
[71] Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayaman, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. IJCV, 2022.
[72] Tianlu Wang. Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In ICCV, 2019.
[73] Zeyu Wang. Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards faimess in visual recognition: Effective strategies for bias mitigation. In CVPR, 2020.
[74] Ross Wightman. GitHub repository: Pytorch image models. GitHub repository, 2019.
13

---

[75] Kaiyu Yang, Klint Qinami, Fei-Fei Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree the imagenet hierarchy. In Conference fairness, accountability and transparency, 2020.
[76] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. In NeurIPS, 2024.
[77] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[78] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representa- tions. In JCML, 2013.
[79] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 2018
[80] Hongyi Zhang, Moustapha Cisse. Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[81] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv: 2017.
[82] Bolci Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep
features for discriminative localization. In CVPR, 2016.
[83] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
14

---

Appendix
Table of Contents
A Implementation Details
16
A.1 Dataset Classification with Transformations
16
A.2 LLM-based Analysis
17
B Additional Results on Dataset Classification with Transformations
19
B.1 Handcrafted Features
19
B.2 Ablation of Different Pre-trained Models
19
B.3 Dataset Classification with Different Vision Backbones
21
B.4
Combining Information from Multiple Transformations
22
B.5 Is the Dataset Classification Model Memorizing or Generalizing?
22
B.6 High-pass and Low-pass Filters at Different Thresholds
23
B.7
Representing Semantic Segmentation Masks with Binary Arrays
24
Verifying Semantic Patterns
C.1 Vision-Language Models
24
C.2 VisDiff
25
Examples of Long Captions
26
E Additional Information ImageNet Object Analysis
27
Limitations
27
G Broader Impacts
27
15

---

A Implementation Details
A.1 Dataset Classification with Transformations
Dataset classification training. Following Liu and He [40], for image-text datasets CC and Data- Comp, we only their images. For each dataset in the YCD combination, uniformly sample 1M/ 10K to form the train val sets. To speed up image loading, the shorter side of each image is resized 500 pixels if the original shorter side larger than this, with the aspect ratio preserved. We observe that this resizing has minimal effect on model performance. The dataset classification model trained for 30 epochs, 23% of the original training length in Liu and He [40]. Table details our default training recipe for dataset classification in Section 3.
config
value
optimizer learning rate weight decay
AdamW [42]
1c-3
0.3
optimizer momentum
B1, 0.95
batch size
4096
learning rate schedule
cosine decay
warmup epochs
training epochs
30
augmentation
RandonResizedCrop
[65] & RandAug (9, 0.5) [13]
label smoothing mixup [80]
0.1
0.8
cutmix [77]
1.0
Table 1: Training recipe for dataset classification.
During training, the model receives randomly augmented 224 image crops as input. At inference time, each image is first resized so that its shorter side has 256 pixels, with the aspect ratio maintained. 224x222 center crop is then extracted and used as the model's input.
Data processing. By default, we use BandomResizedCrop and RandAug as data augmentations in training. Nevertheless, the data augmentations may need adjustments for certain image transforma- tions. Specifically, we have the following two design choices: (1) We apply most transformations before data augmentations to avoid the time cost of transforming every augmented image. However, for pixel / patch shuffling and low-pass/ high-pass filter, we apply these transformations after data augmentations. This because patch sizes in patch shuffling often cannot evenly divide the original
transformation
transformation before data augmentation
use RandAug
original
N/A
semantic segmentation
object detection
VAE
Canny edge detector
SAM contour
depth
pixel shuftling
X
patch shuffling
X
RGB mean
low-pass filter
high-pass filter
unconditional generation
text-conditional generation
SIFT
X
HOG
X
Table 2: Data augmentation details for classifying transformed images.
16

---

image dimensions. Moreover, shuffling with a fixed patch size on original images of varying dimen- sions before augmentations leads to inconsistent patch sizes the final 224 x 224 augmented images. For high-pass/ / low-pass filter, image augmentations alter the frequency of visual signals, so applying these filters after augmentations ensures the intended frequency range is preserved. (2) RandAug is not used for images transformed into binary or grayscale formats (e.g., Canny edge detector and low-pass filter) and those represented by color palettes (e.g., semantic segmentation). This because brightness and contrast adjustments in BandAug are designed for standard RGB images. Table 2 shows whether we apply transformation before data augmentation and whether we use RandAug.
Other details. In Section 3.2, we use LLaVA 1.5 [39, 38] with 4-bit quantization [16] for image captioning. The quantization minimally degrades performance. To perform the dataset classification on the generated captions, we finetune the Sentence TS-base [50] model using a batch size of 128. We search over learning rates ( (1e-3, le-4, le-5) and numbers of training epochs {1, 2, 4, 6). The number of warmup iterations is set to 6% of the total training iterations. For VAE, we use the Kl-regularized VAE [58] with a downsampling factor of 4, which encodes an RGB image of shape 256x256x3 into latent vector of size x x4 3. In Section 3.7, we train DiT-B/2 [52] model each of YCD datasets for 275K iterations, using a batch size 1024 and a constant learning rate le-4.
A.2 LLM-based Analysis
In Section 4.2, we leverage LLMs to perform dataset classification with in-context learning on LLa VA- generated captions of YCD datasets and summarize the characteristics of each dataset. GPT-4o is the default LLM in our analysis. We accessed via ChatGPT Temporary Chat in May 2024.
Here are some samples from three different distributions of captions.
(Caption 1) is from distribution (1, 2, or 3)
(Caption 2) is from distribution (1, 2, or 3)
(Caption 360) is from distribution (1, 2, 3)
Determine which of the three distributions is this caption sampled from: (validation caption)
Figure 17: In-context learning prompt. We provide an LLM with 360 in-context demonstrations, comprising 120 captions sampled from each of the YCD datasets. The model is then prompted it to predict the dataset origin of a hold-out caption. To prevent the LLM from using any prior knowledge of the YCD datasets, each dataset is anonymized as "distribution 1
In-context learning. Figure 17 shows our in-context learning prompt. We create 360 in-context examples by sampling 120 long captions from each of the YCD datasets. Based these in-context examples, the LLM needs to determine the source dataset of a hold-out caption. To avoid the LLM using any prior knowledge about the YCD datasets, we randomly assign index from {1, 2, 3) to represent each dataset. We repeat the entire process with different in-context examples and evaluate the classification accuracy averaged across different hold-out samples. As shown in Figure 18, the accuracy stabilizes as the number of samples from each dataset reaches 160.
0.7
accuracy (%)
0.6
0.5
0.4
in-context learning
chance level
0.3
40
80
120
160
number of hold-out samples per dataset
Figure 18: The accuracy in-context learning converges at 160 samples per dataset.
Summarization for dataset characteristics. We employ a two-step process for an L.LM to sum- marize the unique characteristics of each dataset. Figure 19 illustrates our two-step procedure and
17

---

the prompts. First, we provide the LLM with combination of 360 randomly sampled captions (120 per dataset) and ask it to identify 2 distinctive patterns for each dataset. We repeat this process 10 times with different samples, generating 20 patterns for each dataset. Second, the LLM condenses the 20 patterns into 5 bullet points that characterize each dataset. Figure 20 shows the full L.LM summarization of each dataset's characteristics.
Here some samples from three different distributions of captions.
(Caption 1) from distribution (1. 2. 3}
(Caption 2) from distribution (1. 2.
unique characteristics for each distribution
(Caption 360) is from distribution (1, 2, or 3)
Determine unique characteristics the three distributions with two concise builet points each. Focus on the content rather than textual features.
x10 times
for each of YFCC, CC, and DataComp:
Below 20 bullet points summarizing the unique characteristics of distribution of images. Remove the redundant and combine them into 5 bullet points focusing visual elements, each consisting of one title and short sentence.
summarized dataset features
(Bullet point 1}
(Bullet point 5}
Figure 19: Prompting procedure for LLM summarization. The LLM summarizes the dataset char- acteristics over 10 iterations, with 360 randomly sampled captions per iteration. These characteristics are then aggregated, and the LLM consolidates them into five bullet points for each dataset.
YFCC
CC
DataComp
1. Group Dynamics and Activities
1. Organized Indoor and Outdoor Scenes
Object-Focused Descriptions
This distribution frequently showcases groups of people engaged in activiles such playing music, attending events, or participating in sports, emphasizing social interactions communal seltings.
Captions prominently feature specific objects or products (e.g., coffee mugs, toys, cars), aften isolated against minimalistic backgrounds to highlight their characteristics.
Captions depict well-structured environments, including cazy bedrooms, dining areas, cityscapes, architectural landmarks, emphasizing the arrangement and detail.
and Captions often describe dynamic
Settings
Vibrant and Playful Visuals
environments with people activity in urban or public settings, such as busy city transportation hubs, events.
Human Interactions and Social Events
Scenes frequently include vibrant, colorful, and playful elements, focusing visually appealing and lively imagery that captures attention.
Emphasis gatherings like weddings, concerts, and ceremonies, highlighling attire, decor, and the lively atmosphere.
3. Serane Natural Settings
3. Close-Up and Detalled Views Descriptions often emphasize close-up highlighting the intricate details, textures, and designs of objects, with a focus on aesthetic and functional attributes.
Many images feature environments, including natural landscapes, gardens, and bodies of water, highlighling a calm and peaceful atmasphere.
ouldoor
Vivid Dynamic Elements Descriptions focus on colorful and lively soenes, with vibrant attire, festive settings, and active engagements, emphasizing visual appeal and movement.
4. Detailed Environmental Context Descriptions typically include detailed alements of the scene, focusing on settings trees, buildings, objects natural urban environments. emphasizing the rich
Serene and Artistic Compositions Many images showcase static, picturesque environments or designs, creating a calm and detailed visual representation of both natural structured settings.
Detailed Objects Clothing
Caplions highlight specific objects and items of clothing, such as jewelty and accessories, with intricate details patterns, colors, and designs.
interactions between elements.
5. Emotions and Interactions
Simplistic and Isolated Backgrounds
frequently capture celebration, competition, or personal interaction, highlighting emotions dynamic aspects of the subjects in various social and communal settings.
5. Creative and Artistic Themes
presence scenes including paintings, murals. sculptures, and elements of performance and fantasy, showcasing creativity.
against simplistic, plain backgrounds, drawing attention the main subject and creating a clean, uncluttered look.
Scenes
Figure 20: Full list of L.LM summarization of dataset features.
18

---

B Additional Results on Dataset Classification with Transformations
B.1 Handcrafted Features
In Sections 3.2 and 3.3, we primarily leverage modern neural networks to transform images into alternative representations. Here, we apply classic computer vision algorithms to extract handcrafted features and assess whether these features can effectively capture dataset bias.
YFCC
CC
DataComp
SIFT
65.2%
1:14
HOG
79.9%
+-0.%%
Transformations encoding handcrafted features. SIFT keypoints can moderately capture dataset bias, while HOG achieves close-to-reference accuracy by capturing local gradients.
Figure 21:
Scale-Invariant Feature Transform (SIFT) [43] identifies keypoints that are invariant scale changes. Each keypoint is characterized by location, scale, and orientation. SIFT computes the Difference of Gaussian (DoG) across multiple scales and identifies keypoints as the local extrema in the DoG images. These keypoints are then refined through Taylor expansion for precise localization, and orientation is assigned based on local gradient directions. In Figure 21, we visualize each keypoint eircle, with radius length and direction indicating its size and orientation. The classification accuracy of 65.2% indicates that SIFT features are moderately effective at capturing dataset bias.
Histograms of Oriented Gradients (HOG) [14] robustly represents gradient patterns by aggregating gradient information within small spatial regions (cells) into histograms. These histograms are normalized over block of adjacent cells for better invariance to changes in illumination and contrast. Figure 21 visualizes HOG with the dominant gradient direction in each cell, where brightness indicates gradient magnitude. The near-refcrence accuracy of 79.9% suggests that the shape and contour features captured by HOG are highly indicative of the dataset identity.
B.2 Ablation of Different Pre-trained Models
Here we explore how the performance of different pre-trained models used for transformations affects our results in Section 3. Specifically, we perform object detection, contour extraction, and depth estimation using smaller and less powerful models ViTDet-Base. SAM (ViT-Base), and Depth- Anything- V2 (ViT-Base). As shown in Table 3, the pre-trained model size minimally impacts the classification accuracy.
ViTDet-B
ViTDet-H
SAM (ViT-B)
SAM (VIT-L)
parameters
145M
695M
parameters
94M
312M
accuracy
60.8%
61.5%
accuracy
72.3%
73.1%
Depth-Anything V2 (ViT-B) Depth Aooth-Anydhing-222 (VIT-L)
parameters
98M
335M
accuracy
72.6%
73.4%
Table 3: Varying pre-trained model size for object bounding box generation, SAM contour forma- tion, and depth estimation minimally affects dataset classification accuracy on transformed datasets.
19

---

Additionally, we apply our LLM summarization method in Section 4.2 to other LLMs: Claude 3.5-Sonnet [2] and Llama-3.1-88 1-8B-Instruct [17]. Figures 22 and 23 list the summarized dataset features. The features from different LLMs express highly similar concepts for each dataset.
1. Outdoor Settings
Detailed Focus Emphasizes specific objects, people, or scenes technical details close-up views.
Object Focus
Predominantly features natural landscapes, cityscapes, and public spaces. showcasing diverse environments from beaches and forests to urban streets.
Close-up views of individual products or simple scenes dominate the imagery.
2. Group Dynamics
2. Artistic Representation
2. Background
Frequently depicts large gatherings and crowded scenes, capturing social interactions and collective activitles in various contexts.
Frequently includes stylized or creative designs, from logos to cartoon characters and abstract compositions.
Items are frequently presented against plain, often white backgrounds for dlarity,
3. Action and Movement
3. Clothing and Accessories
3. Branding
Ofien portrays dynamic scenes with people animals motion, emphasizing the energy and liveliness of the captured moments.
Often provides detailed descriptions of attire, personal Items, and their positioning within the image.
Product images commonly feature visible brand names, logos, or technical specifications.
Wildlife and Nature
Setting Variety
Detail Emphasis
Regularly includes flora and fauna elements, highlighting the presence animals and natural features within the images.
Covers diverse enviroommnts, from intricate layouts to historical scenes and unique scenarios.
Descriptions highlight specific features, materials, colors, and designs of objects.
Visual Details
5. Content Varlety
Provides rich descriptions of clothing, surroundings. and contextual elements, offering a comprehensive view of the scenes depicted.
Text and Branding
Imagery includes diverse elements like household items, abstract designs, and Informational content such as diagrams or book covers.
Commonly features written elements, references to specific brands, characters, pop cullure icons.
Figure 22: Claude 3.5-Sonnet's summarization of dataset features.
1. Natural Environments
1. Detailed Compositions The images oflen feature detailed and composed scenes, with an emphasis on showcasing the textures and details of the subject matter.
1. Focus on Color and Texture The images often feature focus color, shape, or texture, particularly in abstract conceptual images, and detailed close- views of objects or visual elements.
The distribution features images of serene and picturesque natural environments, such landscapes, forests, and mountains.
2. Minimalist Backgrounds
2. Everyday Life and Human Connection
2. Urban Landscapes and Settings The images frequently depict urban environments, such streets, buildings, and cityscapes, capturing the energy and activity city life.
The images frequently have a more minimalist neutral background, which allows the object or product to be the main focus, often with a focus on showcasing their design, texture, and visual appeal.
The images depict people engaging varlous activitles, creating a sense of everyday life human connection in natural or everyday settings.
3. Stylized and Artistic Visuals
3. Object-Centric Compositions
Vibrant Colors and Dynamic Compositions
The images often have more stylized or
The images often feature objects, products, or scenes that are more static or still, such product background, diagram or chart, or scenic view, with focus on showcasing their design, texture, and visual appeal.
artistic quality, with focus composition, lighting. and texture, and may include text, logos, other visual elements that add context or meaning to the scene.
The images often feature vibrant colors, dynamic compositions, and sense of energy or movement, capturing the beauty and excitement of the world around us.
white
4. Focus on People and Relationships
4. Emphasis Objects and Concepts
4. Attention to Detail
The images frequently include people as the main subject, with focus their actions, emotions, or interactions with their environment, often conveying a sense of energy, movement, or excilement.
The images frequently focus on objects, concepts, or ideas, highlighting their significance, functionality, or emotional resonance, often with mix of objects. people, and text.
The images frequently include detailed, close-up views of objects visual elements, such as intricate designs, patterns, textures, which suggests focus on craftsmanship and artistry.
5. Visual Arrangement and Composition
5. Simple yet Effective Composition
5. Realistic and Detalled Scenes The images often feature everyday scenes, presented in a realistic and detailed manner, showcasing natural environments and ordinary activilies.
The images often feature focus simple colors, clean backgrounds, and focus main subject or information being conveyed, resulting in sense of simplicity and minimalism.
The images often feature objects, food, or other visual elements arranged in a
visually appealing manner, with focus color, texture, or composition.
Figure 23: 1i-ma-3.1-8B-Instaaee summarization of dataset features.
20

---

B.3 Dataset Classification with Different Vision Backbones
In Section 3, we employ ConvNeXt- Tiny as the base classifier to perform the dataset classification task transformed images. To see how our results change for models with different sizes and architectures, we further train ConvNeXt--Fetto [74], ConvNeXt-Nano, and ResNet-34 [26] to perform the dataset classification task for each transformation. In Table 4, the results show that the classification accuracy for each transformation stable across different models.
ConwNeXII-Femto
ConvNeXI-Nano
ResNet-34
ConvNeXt-Tiny
parameters
4.8M
15.0M
21.3M
27.8M
reference
79.9%
81.2%
81.6%
81.7%
semantic segmentation
68.8%
69.8%
71.7%
70.3%
object detection
59.8%
60.1%
61.7%
61.5%
VAE
73.7%
76.2%
77.3%
77.1%
Canny edge detector
69.2%
70.3%
71.1%
70.8%
SAM contour
70.6%
72.3%
73.8%
73.1%
pixel shuffling (random order)
52.6%
52.6%
60.8%
52.2%
pixel shuffling (fixed order)
56.6%
57.7%
59.5%
58.5%
patch shuffling (random order)
78.5%
78.7%
79.2%
80.2%
patch shuffling (fixed order)
78.5%
79.2%
79.9%
81.0%
RGB mean
47.9%
47.9%
48.0%
47.9%
low-pass filter"
71.2%
72.7%
70.4%
73.8%
high-pass filter*
76.6%
78.9%
81.6%
79.1%
unconditional generation
77.1%
79.9%
81.6%
76.5%
text-conditional generation
56.8%
57.3%
57.9%
57.8%
Table Different image classification models' validation accuracy on transformed datasets. The accuracy remains consistent across models and transformations. Note the frequency filters here are Butterworth filters [9] with a threshold
We also finetune two additional sentence embedding models MPNet-Base [63] and Sentence-BEr
Base [57] for dataset classification on LLa LLaVA-generated captions for YCD images. Note both of these models are weaker [49] than our default Sentence-TT T5-eeseT in Section 3.2. Nevertheless, as shown in Table 5, the classification accuracy remains consistent across models and transformations.
generated
transformation
MPNet-Base
Sentence-BERT BERT-Base
Sentence- ----rc------Beeee
short caption
63.6%
63.4%
63.7%
long caption
66.2%
65.9%
66.0%
Table 5: Different sentence embedding models' dataset classification accuracy on generated captions of YCD images. The accuracy is still high even on weaker sentence embedding models.
21

---

B.4 Combining Information from Multiple Transformations
We are interested in whether combining several visual attributes can jointly contribute larger dataset bias. To this end, we transform each image in the YCD datasets using two different transfor- mations, and concatenate the two resulting images along the channel dimension into single image. We consider all pairwise combinations of object detection, pixel shuffling, and SAM contour. Table 6 shows the resulting classification accuracies. Combining semantic and structural attributes can result in higher dataset classification accuracy compared to using single attribute alone.
transformation (accuracy)
transformation (accuracy)
combined accuracy
pixel shuffling (52.2%%
object detection (61.5%)
68.9%
pixel shuffling (52.2%)
SAM contour (73.1%)
74.2%
object detection (61.5%)
SAM contour (73.1%)
73.1%
Table 6: Combination of different transformations can lead larger bias.
B.5 Is the Dataset Classification Model Memorizing or Generalizing?
To show that the high validation accuracy on the transformed datasets Section is achieved through generalization rather than memorization of training examples, we follow Liu and He [40] to perform the dataset classification task on pseudo-datasets. However, unlike Liu and He [40], construct the pseudo-datasets from transformed images instead of original images.
Specifically, create three dataset applied with one specific transformation. Tables and 8 present the pseudo-datase clas- sification training accuracy YFCC bounding boxes and YFCC Canny edges, without with augmentations. As expected, the classification models fail to converge with more training images stronger augmentations. All pseudo-dataset classification modely have chance-level validation accuracy of 33%, as they merely memorize the dataset origin of each training image rather than learning generalizable patterns.
datasets, each sampled without replacement from the YFCC
imgs per set
w/o aug
w/ aug
imgs per w/o aug
aug
100
100%
100%
100%
100%
1K
100%
100%
IK
100%
100%
10K
100%
fail
10K
100%
fail
100K
fail
fail
100K
fail
fail
Table 7: Training accuracy for YFCC bound- ing box pseudo-dataset classification.
Table 8: Training accuracy for YFCC Canny edge pseudo-dataset classification.
22

---

B.6 High-pass and Low-pass Filters at Different Thresholds
low-
pass
filter
high-
pass
filter
35
65
95
125
155
Figure 24: Ideal filter [22] with different thresholds. We select filtering thresholds (5, 35, 65, 95, 125, 155). The high-pass filtered images are equalized for better visualization.
low-
pass
filter
high-
pass
filter
125
155
Figure 25: Butterworth filter [9] with different thresholds. We select filtering thresholds (5, 35,
65, 95, 125, 155]. The high-pass filtered images are equalized for better visualization.
23

---

In Section 3.6, we perform dataset classification on high-pass and low-pass filtered images to quantify the dataset bias in different frequency components. Here, we further investigate how varying thresholds affect classification performance. Figures 24 and 25 display transformed images with ideal filter [22] and Butterworth filter [9]. Figure 26 presents the resulting dataset classification accuracies across different thresholds. While accuracies generally decline at very strict threshold values, in other cases, ideal and Butterworth filters achieve high accuracies for both low-pass and high-pass filters.
80
accuracy
accuracy
60
75
low-pass
low-pass
high-pass
high-pass
40
70
reference
reference
20
10
50
90
130
10
50
130
filter threshold (ideal)
filter threshold (Butterworth)
Figure 26: Accuracy on filtered images at different thresholds. For most thresholds, both ideal and Butterworth filters achieve high accuracies both high-pass and low-pass filters.
worth noting that the Butterworth filter employs a soft thresholding approach when filtering frequencies, therefore allowing each image to retain some frequency information beyond the selected threshold. This likely contributes to the high accuracy observed on Butterworth high-pass filtered images, even at higher thresholds. Additionally, a higher classification accuracy with a threshold of 150 compared to 115. This possibly because when the threshold is at 1 115, the biased freeuferyyenccy information.
observe that the ideal high-pass filter can achieve
information is dominated by less-biased
B.7 Representing Semantic Segmentation Masks with Binary Arrays
In Section 3.2, apply semantic segmentation to capture semantic information from images and perform datasct classification on the segmentation masks. In particular, we represent the 150-class semantic segmentation mask assigned a distinct color from the predefined palette. This results 69.8% accuracy. Alterna- tively, training the classification model directly the 150-channel binary segmentation mask yields a slightly lower accuracy of 67.6%. The difference in performance may result from the model processing the compact RGB format effectively than the sparse, 150-channel binary format.
RGB image using color palette, where each semantic class
C Verifying Semantic Patterns
In Section 4.2, leverage various language analyses provide natural language descriptions of each dataset's characteristics. Here, we aim to validate these characteristics with a Vision-Language Model (VLM) and VisDiff [18].
C.1 Vision-Language Models
this product showcase?
Does this image have entirely white background?
Is there any person image?
Is this image depicting an outdoor scene?
Is this image depicting domesiic scene?
CC
DataComp
100
d0
d
50
100
percentage (%)
Figure 27: High-level semantic features' distributions annotated by LLaVA. The imbalanced distributions across YCD confirm the dataset characteristics in Section 4.2.

---

To confirm the semantic patterns we found in the YCD datasets, we pick five representative ones and prompt a VLM LLaVA 1.5 [38] to answer whether each pattern exists in imgges. In Figure 27, we plot the distribution of Va's responses across the YCD datasets. The results quantitatively substantiate the identified dataset themes, further evidencing that (1) DataComp characterized by product showcase and white backgrounds but lacks human figures and (2) YFCC focuses on outdoor scenes, while CC and DataComp contains more images depicting domestic and indoor environments.
We also manually check the VLaVA annotation quality with handful of samples. Table 9 displays some examples. The question and images involving human presence are omitted for privacy reasons.
dataset
YFCC
CC
DataComp
image
product
X
showcase
white
X
X
x
background
outdoor scene
domestic X
V
X
V
X
x
X
X
scene
Table 9: Examples of LLaVA annotations. LLaVA can answer our question with reasonable accuracy. The four rows correspond to the first, second, fourth, and fifth questions in Figure 27.
C.2
VisDiff
YFCC
DataComp
unique
product Images images of beauty products
(compared to) YFCC
pictures with elements
food cooking
furniture and interiors
clothing or fashion items
outdoor sports activities
furniture and appliances
(compared to) CC
Interactions between people
Images sports goods
objects in black and white photography
outdoor recreational activities
people involved activities
(compared to) DataComp
people gatherings
Interactions between people
human focused Images
performing tasks
people pasing for pholos
Figure 28:
Dataset features generated by VisDiff [18]. Each cell lists the top 3 concepts distinguish- ing the dataset in each column from the one in each row. Note the VisDiff concepts highly overlap with the dataset characteristics from our language analysis Section 4.2.
VisDiff [18] is recently proposed method for describing differences between two image sets in natural language. first uses a VLM to generate image captions, and then leverages an LLM to propose concepts that distinguish the first image set from the second image set based on these captions. For each concept, VisDiff uses the CLIP similarity scores between the images and that concept to classify two image sets, and calculates the AUROC of this classification. The final output concepts are ranked based on their corresponding AUROC scores.
For each dataset pair in the YCD datasets, we use VisDiff to identify the top 5 concepts that better describe the first dataset compared the second one, as shown Figure 28. VisDiff's concepts closely align with the dataset characteristics of our language analysis in Section 4.2, highlighting "people" and "outdoor activities" for YFCC and "products" for DataComp. However, our method can also identify specific visual biases across datasets with various transformations, fixed object-level queries, and in-depth natural language analysis.
25

---

D Examples of Long Captions
Table 10 shows examples of long paragraph captions generated by LLaVA 1.5 [38] in Section 3.2.
dataset
images
long paragraph caption
A black and white dog standing in shallow body of water, possibly river stream. dog appears to be drinking from the stream, surrounded by rocks. The scene captures the dog's s natural behavior as it quenches its thirst at water source.
YFCC
A food truck parked on side nearby. The truck is blue and features surfhoard on the side, indi- cating surf theme for its offerings. The man, possibly the owner or employee, stands beside the truck, either waiting for customers preparing to serve food.
road, with a man standing
A large herd of elephants is walking across grassy field, with at least 13 elephants visible. Some closer the foreground, while others are further in the background, spread out across the field pairs or individually. The scene captures the majesty of these creatures in their natural habitat.
CC
A cozy bedroom featuring bed, desk, chair. Above the desk, TV is mounted on the wall, and a laptop rests on the desk, accom- panied by vase of flowers that adds elegance to the room. The space also includes dining table and chairs, providing versatility for different activities.
A spacious dining room with a large wooden table surrounded by chairs. The table set with a white tablecloth, and pillows on the chairs add to the cozy ambiance. Natural light fills the room, and potted plants placed around the area, enhancing the overall atmosphere.
DataComp
A notebook and pen are placed on a table next to a lollipop. The notebook, open with polka dot design, lies beside the pen, ready for use. The colorful lollipop adds playful element the scene.
Table 10: Examples of long paragraph captions generated from LLaVA 1.5 [38).
26

---

E
Additional Information on ImageNet Object Analysis
100
accuracy per class
90
80
70
40
50
60
70
80
90
majority dataset share (%)
Figure 29: Majority dataset share for each ImageNet object class positively correlates with the reference dataset classification accuracy on YCD images within that object class.
The difference in object distribution among datasets based on ImageNet object queries in Section 4.1 can be further used to perform dataset classification. Specifically, we classify an image into the dataset that has the highest frequency of this image's label. Without any learnable parameters (unlike logistic regression in Section 4.1,, this simple decision rule achieves validation accuracy of 50.41 %.
The dataset classification model trained original YCD datasets in Section 3.1 might also leverage the imbalance of underlying object-level distributions. To investigate this, we partition the YCD images by their ImageNet object annotations in Section 4.1. For each object class, we calculate the proportion of its images originating from YFCC, CC, and DataComp, respectively, and define the highest proportion as the majority dataset share. Further, we calculate the dataset classification accuracy for images with each ImageNet object annotation. Figure 29 shows that the dataset classification accuracy for each object class is positively correlated with its majority dataset share.
F
Limitations
Since our framework relies heavily pre-trained recognition and generative models [58, 12, 36, 34, 39] for extracting semantic and structural information from images, the analysis may be affected by the bias inherent to those models or the datasets they trained on. For example, if the pre-trained models are trained on data very similar to certain dataset under study (e.g., one of YCD), the measured level of bias may be affected. In addition, dataset classification can only reveal bias by comparing multiple datasets, and can not be directly applied to a single dataset to understand its bias.
G Broader Impacts
Our framework can be used analyze the datasets before model training, to better determine whether the dataset composition aligns with training goals. It is relatively a fast process compared to a full training cycle. This help researchers reduce experiment iterations and thus total energy usage.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have accurately made claims both in the abstract and introduction to reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have included all details about training with transformed inputs Ap- pendix A.I and LLM's in-context learning in Appendix A.2.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification:
We will release our code before the conference date.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: futtps://nips.. cc/
http://ggiide//odeuuieeeee httpibimssssooooooiiioooooooeeeo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
hatpsccccublccccc /etoedddee///oouummiiiiii /Cc////odeessssseeeeeeeeeee onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See Question
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We perform our experiments Section 3 random samples of 3M training images and 30K validation images, and report the mean validation accuracy and standard deviation. For in-context learning in Section 4.2, GPT-4o is provided with 360 randomly sampled captions from the train set and evaluated on validation sample each time. We sample total of 480 validation samples until accuracy stabilizes, as shown in Figure 18.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details about compute resources here. We use 8 NVIDIA 2080 Ti to train the ConvNeXt-T model for the dataset classification task with 8 gradient accumulation steps. The average compute time for each experiment is about 1.5 days.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / htt./iuuur.pppeee cc it
Answer: [Yes]
Justification:
We have followed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the papers for all datasets we leverage in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi wwwode.hho....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
32

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "YiO6LL3i3j", "reviewer": "Reviewer_yVzd", "summary": "This paper studies the problem of dataset bias prevalent in current multimodal datasets. It revisits the dataset classification experiment from Torralba et al, recently studied again by Liu and He, and deconstructs their findings to understand what aspects of datasets (structural, semantic, color, object-level, caption-level etc) contribute most to the bias prevalent in visual datasets. With abundant empirical evidence, the paper provides an interesting and important analysis on the fundamental gaps in our understanding of large-scale image-text datasets.", "questions": "All my questions are also mentioned in the weaknesses section above.", "limitations": "Yes the authors have mentioned that their main weakness is with respect to using pretrained models which themselves might be biased.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written and presented, the research question is concisely stated, and all the experimental results clearly tie back with the main research question.\n- The research question studied in itself is on that is pivotal in the current age of foundation models. Understanding datasets is key for understand models, and this paper takes a few steps to further our understanding of these giant datasets.", "weaknesses": "I have a few technical concerns that might undermine the significance of this paper's findings, I note them down here. Further, I also have a few suggested additional experiments that might help boost the significance of this paper's results.\n\n- All the results rely on a three class classification problem at heart. How should one interpret these results under the argument that neural networks in general can learn noisy labels, and can fit arbitrary distributions [1]? How does the baseline 81.7% performance change if you assigned random labels to the original set of 3M images? This is a very important ablation to verify the significance of all the findings.\n\n- One issue with the semantic segmentation classification experiment (sec 3.2) is that it removes the pixel information from the task, and converts it into a potentially much simpler task where the input features are only of size 150 (if I understand the experiment correctly). Here is a simple suggestion for removing that confounder. What happens if you train directly on the semantic masks itself? Of-course there would be some bias coming in from the colour applied to the segmentation masks for each of the different object classes, but this could be mitigated by training two different models using two different mask colour palettes, and looking at the variance. If the accuracy of this particular task still remains similarly high, that would further boost the significance of the results.\n\n- For the caption classification task (sec 3.2), could you also use another sentence embedding model that is potentially weaker in its initial MTEB [2] performance, for reproducing the results of that experiment. Again, this would help boost the significance of the findings. \n\n- Would the main results be consistent across two completely different subsets of the YCD classificaiton task? Could you sample an entirely different set of 3M images and 30K validation images, rerun the experiments, and cross-validate on the two different validation sets?\n\n- For figure 11, could you redo that analysis on another subset of 3M samples as well? I am skeptical that this result would hold exactly true for a given dataset, especially given this prior result that most datasets curated from the web have similar concept distributions [3]. Can the authors provide a reconciliation between this prior result and their results?\n\n- Another interesting analysis that can be done would be to use the VisDiff tool [4] to perform an analysis on the visual distribution differences between the pairs of YCD datasets. It might provide an interesting sub-analysis and would help further verify the results in sec 4.2 and fig. 16 in the paper.\n\n- The paper has a lot of very interesting insights about bias across visual datasets. However, there are no concrete suggestions/discussion on how one could go about mitigating these biases. A discussion regarding the source of these datasets, different data curation / filtering mechanisms, and how they might potentially impact these biases, would make for a great and required addition to this paper.\n\n[1] Zhang et al, Understanding deep learning requires rethinking generalization\n[2] Muenighoff et al, MTEB: Massive Text Embedding Benchmark\n[3] Udandarao et al, No \"Zero-Shot\" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance\n[4] Dunlap et al, Describing Differences in Image Sets with Natural Language", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comment. We are encouraged that you find our paper provides helpful insights about bias in large-scale datasets. We address your concerns below:\n>w1: How should one interpret these results under the argument that neural networks in general can learn noisy labels, and can fit arbitrary distributions? How does the baseline 81.7% performance change if you assign random labels to the original set of 3M images?\n\nAll the reported accuracies in our work are evaluated on 30k validation samples. On the other hand, **the findings in [1] about high model accuracy on random labels through memorization are only on the _training_ set.** When we assign random labels to our data, the model only achieves a chance-level accuracy of 33.3% on the validation set. This is expected since models trained on random labels cannot learn dataset-specific patterns that is generalizable to validation sets.\n>w2: What happens if you train directly on the semantic masks itself? \n\n**We would like to clarify that we converted each pixel, rather than the full image, to a 150-channel binary array** (Line 79). The transformed image maintains the original width and height.\n\nAdditionally, training on the _RGB_ segmentation masks with 2 different color palettes results in stable accuracies of 70.1% and 70.0%.\n>w3: For the caption classification task (sec 3.2), could you also use another sentence embedding model that is potentially weaker in its initial MTEB performance?\n\nWe used weaker models MPNet-Base [2] and Sentence-BERT-Base [3] for caption classification:\n||MPNet|Sentence-BERT|Sentence-T5|\n|-|-|-|-|\n|Short Cap.|63.2%|63.2%|63.7%|\n|Long Cap.|64.5%|65.3%|66.0%| \n\n**The accuracy is consistent on all 3 sentence embedding models.**\n>w4: Could you sample an entirely different set of 3M images and 30K validation images, rerun the experiments, and cross-validate on the two different validation sets?\n\nWe reran the experiments on new samples (3M training, 30K validation). Due to time constraints, caption and segmentation experiments are on 300k new training samples. **Accuracies vary minimally across two samples.**\n||baseline|Short Cap. (300k)|Long Cap. (300k)|Sem. Seg. (300k)|Obj. Det.|Patch Shuf.|Canny|High-pass|\n|-|-|-|-|-|-|-|-|-|  \n|train (ori.)>val (ori.)|81.7%|61.5%|63.1%|54.6%|61.5%|80.3%|70.84%|79.4%|\n|train (ori.)>val (new)|82.0%|61.8%|63.5%|55.3%|62.2%|80.3%|70.8%|79.3%|\n|train (new)>val (ori.)|82.1%|61.3%|63.1%|54.6%|61.0%|79.8%|70.4%|80.1%|\n|train (new)>val (new)|82.1%|61.6%|63.6%|55.1%|62.2%|79.9%|71.12%|80.3%|\n\n>w5: For Fig 11, could you redo that analysis on another subset of 3M samples as well? … Can the authors provide a reconciliation between this prior result and their results?\n\nWe observe **the number of overlapped top 10 object categories (as in Fig 11) is high between the original 3M and new 3M images**:\n|#Overlaps|YFCC|CC|DataComp|\n|-|-|-|-|\n|ImageNet|9|8|8|\n|LVIS|10|10|10|\n|ADE20k|10|9|9|\n\nThis does not contradict [4]. Fig 11 shows the top 10 object classes with the highest proportion of their images from a particular dataset, rather than the most frequent object classes within each dataset. Despite certain object categories being overrepresented in certain datasets, the overall object distribution vectors [4] are highly correlated:\n\n|ADE20k Corr |YFCC|CC|DataComp|\n|-|-|-|-|\n|YFCC|1|0.92 |0.82|\n|CC||1|0.97|\n|DataComp|||1|\n\n|LVIS Corr|YFCC|CC|DataComp|\n|-|-|-|-|\n|YFCC|1|0.90|0.81|\n|CC||1|0.93|\n|DataComp|||1|\n>w6: Another interesting analysis ... would be to use the VisDiff tool to perform an analysis on the visual distribution differences between the pairs of YCD datasets.\n\nThank you for recommending VisDiff [5]. We generated caption-based set differences for each dataset pair (**Figure 2 in the attached pdf**):\n||Y|C|D|\n|-|-|-|-|\n|(compared to) Y||unique home decor|Product Images|\n|(compared to) C|outdoor sports activities||furniture and appliances|\n|(compared to) D|People involved in activities|people at gatherings||\n\n**Results from VisDiff highly overlap with LDA- and LLM-extracted dataset characteristics**, emphasizing “people” and “outdoor activities” for YFCC and “product” for DataComp. We’ve added this to our Appendix.\n>w7: There are no concrete suggestions/discussion on how one could go about mitigating these biases.\n\nOur study provides a general framework for identifying the concrete form of low-level and semantic bias in large-scale datasets. **The identified dataset bias can be used retrospectively to analyze the dataset curation procedure.** For example, on YCD:\n- YFCC contains predominantly outdoor scenes and human interactions.\n\nYFCC samples images solely from Flickr, a platform for user-uploaded photos. As Flickr users primarily share personal photos, landscapes, and social interactions, YFCC images predominantly feature natural scenes and human activities. Moreover, YFCC excludes photos labeled as “screenshots” [6], reinforcing the focus on human-related and natural imagery.\n- DataComp has the lowest number of unique objects per image.\n\nDataComp filters for images with high embedding similarity to ImageNet training examples [7], most of which feature object-centric images. This biases the dataset toward images with lower per-image object diversity.\n- CC and DataComp are significantly brighter and contain more digital graphics and object showcase.\n\nThese datasets are collected from the Internet and feature results from search engines, which prioritize professionally created content like advertisements, infographics, and digital media. This results in a higher prevalence of digital graphics and brighter images, optimized for visual engagement and online presentation.\n\nWe have consolidated the above as a discussion paragraph in our draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CUqLtkBTit", "reviewer": "Reviewer_n4tv", "summary": "This work theorizes and investigates various concrete forms of inter-dataset biases, namely those among YFCC, CC, and DataComp (YCD). The authors analyze such inter-dataset biases in pure visual attributes as well as in semantic attributes using LLM-generated descriptions.", "questions": "Please find my major questions in the Weakness section.\n\nAnother slight concern is about the presentation. Since an ideal debiased dataset should enjoy an equal likelihood to be predicted as one of the anchor datasets (e.g. YCD), maybe the authors should explicitly clarify this important task setting as early as possible in the paper.\n\n(Aug 9th): Updating my overall ratings thanks to the quality response.", "limitations": "The limitations have been sufficiently addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is a fluent read composed of a clear progressive structure. The authors have shown meticulous effort in constructing the investigations as comprehensive as possible.", "weaknesses": "This work extends the scope of [1] and aims to provide specific insights for constructing less biased data collection in future. However, I feel there are several missing evidence that undermine the contributions of this work in its current state:\n\n1. **The authors have only examined the factors of biases individually.** However, I notice that none of the single attributes listed contributes to a higher prediction accuracy over the baseline of using original visual features. Could it be possible that a combination of multiple visual/semantic attributes jointly contributes to the large overall bias? This needs to be verified.\n\n2. **The authors have only investigated one classification model.** According to [1], the inter-dataset overall bias (high prediction accuracy) is observed with multiple classification models as well. So if we use the smaller ResNet-50 model, can we still observe the high accuracy over each individual structural or semantic attribute? I haven't seen the authors ruling out the confounding factor of the classification model size.\n\n3. **The proposed framework consists of indicative bias metrics only relative to YCD.** However, YCD all have their own intra-dataset biases. So when we are attempting to create a debiased dataset in future, *how can we make sure our newly collected data are truly diversified and fair?* Can the authors provide similar evidence over 'Memorization vs. Generalization' as in [1] Sec. 4.2., and verify individual inter-dataset bias attributes with a truly unbiased pseudo-dataset?\n\n[1] A Decade’s Battle on Dataset Bias: Are We There Yet? Liu et al. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and the insightful questions. We would like to address your concerns below.\n\n>w1: Could it be possible that a combination of multiple visual/semantic attributes jointly contributes to the large overall bias? This needs to be verified.\n\nThank you for this suggestion. To verify this, we consider different combinations of 3 transformations with lower dataset classification accuracies: object detection (61.5%), random-order pixel shuffle (52.2%), and SAM contour (73.1%). For each pair of transformations, we combine them by concatenating the resulting transformed images on the channel dimension.\n||Transformation 1 Acc.|Transformation 2 Acc.|Combined Acc.|\n|-|-|-|-|\n|Obj. Det. + Pixel shuffle|61.5%|52.2%|68.9%|\n|Pixel Shuffle + SAM contour|52.2%|73.1%|74.2%|\n|Obj. Det. + SAM contour|61.5%|73.1%|73.1%|\n\nWe observed that **combining semantic and structural attributes resulted in higher dataset classification accuracy compared to using a single attribute alone.**\n>w2: So if we use the smaller ResNet-50 model, can we still observe the high accuracy over each individual structural or semantic attribute?\n\nWe are happy to provide more results on this. We’ve expanded our main experiments to employ smaller vision architectures: ResNet-50 (23M parameters) [1] and ConvNeXT-femto (5M parameters) [2]:\n||baseline|Obj. Det.|SAM Contour|Patch Shuf.|High-pass|\n|-|-|-|-|-|-|\n|ConvNeXt-Tiny (28M)|81.7%|61.5%|73.1%|80.3%|79.4%|\n|ResNet-50 (25M)|82.0%|61.6%|73.6%|80.2%|81.9%|\n|ConvNeXt-Femto (5M)|79.9%|59.8%|70.8%|78.7%|76.7%|\n\nAdditionally, we use two weaker sentence embedding models (MPNet-Base [3] and Sentence-BERT-Base [4]) for caption classification:\n||Short Cap.|Long Cap.|\n|-|-|-|\n|Sentence-T5|63.7%|66.0%|  \n|MPNet|63.2%|64.5%|\n|Sentence-BERT|63.2%|65.3%|\n\n**Across model sizes and architectures, the classification accuracy over each individual structural or semantic attribute remains high.** We’ve added this result to our Appendix.\n\n>w3: So when we are attempting to create a debiased dataset in future, how can we make sure our newly collected data are truly diversified and fair? Can the authors provide similar evidence over 'Memorization vs. Generalization' as in [1] Sec. 4.2., and verify individual inter-dataset bias attributes with a truly unbiased pseudo-dataset?\n\nWe agree that our framework can only identify bias relative to the given combination of datasets (e.g., YCD) rather than their bias to the universal vision world. **Nevertheless, identifying inter-dataset bias can still be helpful in creating a debiased dataset, which is 'truly diversified and fair', in the future.** For example, in the paper, we used LLM to summarize specific characteristics of each dataset. These characteristics can be used retrospectively to analyze the dataset curation procedure. On YCD:\n\n- YFCC contains predominantly outdoor scenes and human interactions.\n\nYFCC samples images solely from Flickr, a platform for user-uploaded photos. As Flickr users primarily share personal photos, landscapes, and social interactions, YFCC images predominantly feature natural scenes and human activities. Moreover, YFCC excludes photos labeled as “screenshots” [5], reinforcing the focus on human-related and natural imagery.\n\n- DataComp has the lowest number of unique objects per image.\n\nDataComp filters for images with high embedding similarity to ImageNet training examples [6], most of which feature object-centric images. While this empirically leads to higher zero-shot performance of downstream CLIP models, it biases the dataset toward images with lower per-image object diversity.\n\n- CC and DataComp are significantly brighter and contain more digital graphics and object showcase.\n\nThese datasets are collected from the Internet and feature results from search engines [6, 7], which prioritize professionally created content like advertisements, infographics, and digital media. This results in a higher prevalence of digital graphics and brighter images, optimized for visual engagement and online presentation.\n\n\n**We observe similar trends over 'Memorization vs. Generalization' as in Sec. 4.2 of [8] on our transformed datasets.** Specifically, we create 3 pseudo-datasets, all of which are sampled without replacement from the same transformed YFCC dataset (e.g., images of Canny-detected edges or images of object bounding boxes). We perform dataset classification on these pseudo-datasets. The tables below present the pseudo-dataset classification _training_ accuracy on YFCC bounding boxes and YFCC Canny-edges. As the number of training images from each dataset increases, the task becomes harder. _All of the models trained on this pseudo-dataset classification have a chance-level accuracy of 33% in the validation set_. This is because they merely memorize the dataset origin of each training image rather than learning any generalizable patterns.\n\n|#Training Images per YFCC Bounding Box Pseudo-Dataset|without augmentation|with augmentation|\n|-|-|-|\n|100|100%|100%|\n|1K|100%|100%|\n|10K|100%|fail|\n|100K|fail|fail|\n\n|#Training Images per YFCC Canny edge Pseudo-Dataset|without augmentation|with augmentation|\n|-|-|-|\n|100|100%|100%|\n|1K|100%|100%|\n|10K|100%|fail|\n|100K|fail|fail|\n\n\n>q1: Another slight concern is about the presentation. Since an ideal debiased dataset should enjoy an equal likelihood to be predicted as one of the anchor datasets (e.g. YCD), maybe the authors should explicitly clarify this important task setting as early as possible in the paper.\n\nWe appreciate your suggestion. We’ve added a clarification that “An ideal unbiased dataset should have a chance-level probability of being predicted as any of the anchor datasets.” in our introduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "37S7fIG4Wr", "reviewer": "Reviewer_aLTB", "summary": "This paper explores dataset biases and introduces a framework to identify the unique visual attributes that differentiate various datasets. The method involves applying a range of transformations to extract semantic, structural, boundary, color, and frequency information from the datasets, evaluating how each type of information contributes to their distinct characteristics.", "questions": "1. how does the accuracy of pre-trained models affect the inferences or findings?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A diverse range of transformations was considered for this study.\n2. Provides a comprehensive study to identify various factors that allow the identification of various biases that help to distinguish between datasets.\n3. The paper is written well.", "weaknesses": "1. Even though the paper identifies various factors that help in distinguishing datasets, it does not provide any information/ takeaways on how this information could be used to 'build more diverse and representative datasets in the future'.I consider the investigative analysis as a good and significant contribution but not commenting on how to utilise these results in a way that would be useful for the community is a significant shortcoming of this paper. The authors could have moved some of the transformations to the appendix and utilise the space for describing 'key takeaways and explaining how &what dataset curators should keep in mind while building datasets.\n2. No information is provided on how the accuracy of pre-trained models affects the inferences or findings. \n3. There is no discussion on how the format of data transformation affects the findings. For instance, converting the dataset into segmentation masks or object detection boundaries could capture more information in the segmentation results. The authors have not addressed the impact of these different transformation formats on their observed findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review and the constructive comments. We would like to address your concerns below.\n>w1: Even though the paper identifies various factors that help in distinguishing datasets, it does not provide any information/ takeaways on how this information could be used to 'build more diverse and representative datasets in the future'.\n\nThank you for your suggestion. We are happy to include more discussion about takeaways for dataset curation.\nOur study provides a general framework for identifying the concrete form of low-level and semantic bias in large-scale datasets. **The identified dataset bias can be used retrospectively to analyze the dataset curation procedure.** For example, on YCD:\n- YFCC contains predominantly outdoor scenes and human interactions.\n\nYFCC samples images solely from Flickr, a platform for user-uploaded photos. As Flickr users primarily share personal photos, landscapes, and social interactions, YFCC images predominantly feature natural scenes and human activities. Moreover, YFCC excludes photos labeled as “screenshots” [1], reinforcing the focus on human-related and natural imagery.\n- DataComp has the lowest number of unique objects per image.\n\nDataComp filters for images with high embedding similarity to ImageNet training examples [2], most of which feature object-centric images. While this empirically leads to higher zero-shot performance of downstream CLIP models, it biases the dataset toward images with lower per-image object diversity.\n- CC and DataComp are significantly brighter and contain more digital graphics and object showcase.\n\nThese datasets are collected from the Internet and feature results from search engines [2, 3], which prioritize professionally created content like advertisements, infographics, and digital media. This results in a higher prevalence of digital graphics and brighter images, optimized for visual engagement and online presentation.\n\n>w2: No information is provided on how the accuracy of pre-trained models affects the inferences or findings.\n\nTo address your concern, we reran some of the experiments in Sections 3 and 4 using different pre-trained models. **While the accuracy of the pre-trained models slightly affects dataset classification accuracy on the transformed datasets, our results and insights remain unchanged.**\n\nFor Section 3, we use (1) a weaker VitDet-Base [4] model to extract bounding boxes, (2) a weaker SAM (ViT-B) [5] model to extract object contours, and (3) a stronger generative model SD-XL [6] to text-conditionally generate images. **Even when using different pre-trained models for transformations, the dataset classification model still achieves high accuracy**, suggesting that semantic differences and object shape variations are important contributors to the bias among YCD.\n||VitDet-Base (new)|VitDet-Huge|\n|-|-|-|\n|#Parameters|145M|695M|\n|Box AP (LVIS) ↑|43.0%|51.5%|\n|Dataset Classification Acc.|60.8%|61.5%|\n\n||SAM (ViT-Base) (new)|SAM (ViT-Large)|\n|-|-|-|\n|#Encoder Parameters|91M|308M|\n|Dataset Classification Acc.|65.9%|67.3%|\n\n(Due to resource constraints, the SAM experiments are on 300K training samples.)\n||SD-2.1|SD-XL (new)|\n|-|-|-|\n|#Parameters|983M|3500M|\n|Generation Performance|low|high|\n|Dataset Classification Acc.|55.1%|57.8%|\n\nFor Section 4, we use Claude 3.5-Sonnet [7] and Llama 3.1-8B [8] to derive patterns for each dataset as in Figure 16. **The dataset features summarized from these two LLMs closely resemble those of the original GPT-4o**. Specifically, for YFCC, the emphasis is on \"people,\" \"outdoor,\" and \"nature,\" while for DataComp, the focus is on \"white background\" and \"object focus.\"\n\nClaude 3.5-Sonnet:\n|YFCC|CC|DataComp|\n|-|-|-|\n|Outdoor Settings|Detailed Focus|Object Focus|\n|Group Dynamics|Artistic Representation|White Background|\n|Action and Movement|Clothing and Accessories|Branding|\n|Wildlife and Nature|Setting Variety|Detail Emphasis|\n|Visual Details|Text and Branding|Content Variety|\n\nLlama 3.1-8B:\n|YFCC|CC|DataComp|\n|-|-|-|\n|Natural Environments|Detailed Compositions|Focus on Color and Texture|\n|Everyday Life and Human Connection|Urban Landscapes and Settings|Minimalist Backgrounds|\n|Vibrant Colors and Dynamic Compositions|Stylized and Artistic Visuals|Object-Centric Compositions|\n|Focus on People and Relationships|Emphasis on Objects and Concepts|Attention to Detail|\n|Realistic and Detailed Scenes|Visual Arrangement and Composition|Simple yet Effective Composition|\n\nWe will conduct more experiments with other segmentation models and object detection models and add the results and discussion to our draft.\n>w3: There is no discussion on how the format of data transformation affects the findings.\n\nThank you for pointing this out. We acknowledge that the impact of the transformed images’ format could benefit from further elaboration and a more focused discussion in Section 3:\n\n- Semantic segmentation and object detection: Semantic segmentation provides fine-grained per-pixel semantic annotation, whereas object detection only captures coarse-grained spatial information through bounding boxes. The lack of detailed spatial information contributes to the lower dataset classification accuracy on bounding boxes (61.5%) compared to segmentation masks (67.5%).\n\n- Caption: Caption discards all spatial information, creating more discriminative semantic representations through natural language. This textual representation is less affected by the low-level and spatial cues in the images.\n\n- Edge detection and SAM contour: Object contour delineates fine-grained object shape and spatial information, while lacking the rich object semantic information present in semantic segmentation masks and bounding boxes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers:

Thanks for all your constructive comments! We hope our response and additional results can address your concerns. Please let us know if you have further questions or comments and we would be more than happy to discuss.

For reviewer yVzd, please note the PDF file attached contains figures in response to weakness 5 and weakness 6.

Best,\
Authors

Meta-review: All reviewers were positive about the paper after the rebuttal. The authors have done substantial new work (experiments, analyses, etc.) in the rebuttal stage, which helped convince the reviewers. AC suggests the authors incorporate them into the revised paper.
Decision: Accept (poster)