Understanding Bias in Large-Scale Visual Datasets

Authors: boya  zeng
yida  yin
zhuang  liu
u c berkeley
meta  fair
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 07:02:54

Abstract: A recent study has shown that large-scale pretraining datasets are very biased: they can be easily classified by modern neural networks. However, the concrete forms of bias among these datasets remain unclear. In this study, we propose a framework to identify the unique visual attributes distinguishing these datasets. Our approach applies various transformations to extract semantic, structural, boundary, color, and frequency information from datasets and assess how much each type of information contributes to their bias. We further decompose their semantic bias with object-level queries, and leverage natural language methods to generate detailed, open-ended descriptions of each dataset's characteristics. Our work aims to help researchers understand the bias in existing large-scale datasets and build more diverse and representative ones in the future. Our project page and code are available at boyazeng.github.io/understand_bias

Main paper: 
 introduction: 
 Recently, Liu and He [39] revisited the "Name That Dataset" experiment introduced by Torralba and Efros [67] in 2011 which highlighted the built-in bias of visual datasets. It is a classification task where each dataset forms a class, and models are trained to predict the dataset origin of each image. The datasets back in 2011 were found to be classified very accurately [67] . Since then, significant efforts have been devoted to creating more diverse, large-scale, and comprehensive visual datasets [15, 36, 34, 59] . Surprisingly, a decade later, the largest and supposedly most diverse datasets (e.g., YFCC [64] , CC [11] , DataComp [18] ) can still be classified with remarkably high accuracy [39] .Although we now know these large-scale datasets are very biased, a lingering question remains: what are the concrete forms of bias among them 1 , that cause them to be easily classified? Understanding the bias among datasets is essential for addressing it and improving dataset diversity and coverage. Creating datasets that more comprehensively represent the real world is challenging yet crucial [63, 54, 30] . Only then, can we build truly general-purpose vision models-models capable of handling different scenarios out of the box, and performing reliably in various real-world situations [29, 28, 23] .To this end, we develop a framework for understanding the concrete forms of bias among datasets. We isolate the semantic, structure, boundary, color, and frequency information through various transformations. For example, transforming an image into a semantic segmentation map preserves semantics while discarding most texture information. We then perform the dataset classification task on the transformed datasets to quantify how each type of information reflects the bias.To pinpoint the semantic bias in datasets, we employ object-level queries and open-ended language analysis. Specifically, we leverage pretrained vision models to identify objects that characterize each dataset. In addition, using a Vision-Language Model (VLM), we generate image captions as surrogate language representations of the images. We then apply topic models and Large Language Models (LLMs) to generate natural language descriptions for each dataset.We apply our framework to three popular large-scale visual datasets: YFCC, CC, and DataComp, following [39] . Surprisingly, after transforming images into various semantic and structure representations (e.g., object bounding boxes and contours), neural networks can still accurately predict their dataset origin. This highlights the bias in semantics and object shapes. Our object-level queries further reveal a discrepancy in object diversity and distribution across the YCD datasets. Lastly, open-ended language analysis indicates that YFCC emphasizes outdoor and natural scenes with human interactions, while DataComp features digital graphics heavily.Our framework operates on images only and does not require any human annotations, making it compatible with any vision dataset. It can be applied in future dataset curation to assess data diversity, and guide the addition of images with various attributes. We hope this work can help researchers identify dataset bias and develop more inclusive and diverse visual datasets.Captions depict well-structured environments, including cozy bedrooms, dining areas, cityscapes, and architectural landmarks, emphasizing the arrangement and detail.Captions prominently feature specific objects or products (e.g., coffee mugs, toys, cars), often isolated against minimalistic backgrounds to highlight their characteristics.This distribution frequently showcases groups of people engaged in activities such as playing music, attending events, or participating in sports, emphasizing social interactions and communal settings.Captions depict well-structured environments, including cozy bedrooms, dining areas, cityscapes, and architectural landmarks, emphasizing the arrangement and detail.Captions prominently feature specific objects or products (e.g., coffee mugs, toys, cars), often isolated against minimalistic backgrounds to highlight their characteristics.This distribution frequently showcases groups of people engaged in activities such as playing music, attending events, or participating in sports, emphasizing social interactions and communal settings.Emphasizes specific objects, people, or scenes with technical details and close-up views.Close-up views of individual products or simple scenes dominate the imagery.Predominantly features natural landscapes, cityscapes, and public spaces, showcasing diverse environments from beaches and forests to urban streets.The images often feature detailed and composed scenes, with an emphasis on showcasing the textures and details of the subject matter.The images often feature a focus on color, shape, or texture, particularly in abstract or conceptual images, and detailed closeup views of objects or visual elements.The distribution features images of serene and picturesque natural environments, such as landscapes, forests, and mountains. 
related work: 
 Dataset classification. The dataset classification problem was originally proposed by Torralba and Efros [67] in 2011. Tommasi et al. [66] later also studied this problem with linear classifiers using pretrained CNN features. In contrast to prior work focusing on labeled smaller-scale datasets with shared object classes, Liu and He [39] recently revisited the dataset classification problem with large-scale, diverse, and potentially less-biased datasets. They demonstrated that modern neural networks are still excellent at capturing bias among these datasets. Our work identifies the exact forms of the bias among datasets beyond dataset classification results.Social bias and fairness. Extensive literature has highlighted that various visual datasets underrepresent certain demographic groups [68, 73, 50] , contain various gender stereotypes [79, 27, 44] , or ignore some geographical regions [61, 73, 69] . These social fairness issues in datasets result in the deployment of flawed models [8, 70, 6] that may produce biased predictions or struggle to generalize well across different domains. Instead of focusing on social fairness in each dataset, we study a different concept of visual representativeness (i.e., coverage of real-world concepts and objects) and understand its differences among datasets. Note Meister et al. [44] also use transformations to isolate different types of information, with a specific focus on gender bias in datasets.There are approaches that can locate the imbalance of object representation within datasets [22, 69] . Dataset re-balancing methods [8, 73] seek to correct these representation imbalances across protected attributes. Algorithmic intervention and regularization, such as adversarial training [76, 77, 43] and domain-independent training [71] , can counteract the propagation of bias and stereotypes in downstream modeling. Note that these prior methods often require ground-truth annotations to identify or mitigate potential bias, while our framework can analyze unlabeled pretraining datasets and provide insights beyond object distribution imbalances.Emphasis on social and formal gatherings like weddings, concerts, and ceremonies, highlighting attire, decor, and the lively atmosphere.Scenes frequently include vibrant, colorful, and playful elements, focusing on visually appealing and lively imagery that captures attention.Captions often describe dynamic environments filled with people and activity in urban or public settings, such as busy city streets, transportation hubs, and social events.Emphasis on social and formal gatherings like weddings, concerts, and ceremonies, highlighting attire, decor, and the lively atmosphere.Scenes frequently include vibrant, colorful, and playful elements, focusing on visually appealing and lively imagery that captures attention.Captions often describe dynamic environments filled with people and activity in urban or public settings, such as busy city streets, transportation hubs, and social events.Frequently includes stylized or creative designs, from logos to cartoon characters and abstract compositions.Items are frequently presented against plain, often white backgrounds for clarity.Frequently depicts large gatherings and crowded scenes, capturing social interactions and collective activities in various contexts.The images frequently depict urban environments, such as streets, buildings, and cityscapes, capturing the energy and activity of city life.The images frequently have a more minimalist or neutral background, which allows the object or product to be the main focus, often with a focus on showcasing their design, texture, and visual appeal.The images depict people engaging in various activities, creating a sense of everyday life and human connection in natural or everyday settings. 
isolating bias with transformations: 
 Although modern neural networks can achieve excellent accuracy in the dataset classification problem [39] , what bias is captured by neural networks remains unclear. To better understand this, we selectively preserve or suppress specific types of information using various transformations. We then train a new model on the transformed datasets to perform the dataset classification task. As a result, its dataset classification performance indicates the level of bias in the extracted information. For example, transforming an image into a depth map captures the spatial geometry while discarding texture, allowing us to assess bias present solely in spatial information.Building on Liu and He [39] , we take YFCC [64] , CC [11] , and DataComp [18] (collectively referred to as "YCD") and study their bias in this work. Figure 1 illustrates some examples.Our training setup is also adapted from [39] . Specifically, we randomly sample 1M and 10K images from each dataset as training and validation sets, respectively. We also employ the same image Figure 1 : Original images. We sample two images from each of YFCC [64] , CC [11] , and DataComp [18] . Dataset classification on the original images has a reference accuracy of 82.0%. classification model ConvNeXt-Tiny [40] and train it for 30 epochs on the combined dataset with 3M samples. Note the original work [39] achieves 84.7% accuracy on the validation set, while we reproduce an accuracy of 82.0% due to shorter training. We refer to this reproduced accuracy as reference accuracy in this paper. For all the image transformations, we use the same base classifier model. We repeat the data sampling and experiments three times and report the mean validation accuracy and the standard deviation. The training settings are detailed in Appendix A.1.To start, we seek to understand how semantically biased the datasets are. Specifically, we extract semantic components from the images with fine-grained, coarse-grained, or no spatial detail. Also, we apply a variational autoencoder, potentially reducing low-level signatures (e.g., color quantization, JPEG compression). Figure 2 shows the transformations and their dataset classification accuracies.Semantic segmentation offers fine-grained semantic annotation with rich object information by labeling each pixel as a class. We take a semantic segmentation model ViT-Adapter-Large [12] (with BEiT-v2 [52] backbone) trained on ADE20K [81] with 150 semantic classes (e.g., wall, building, sky, etc.) to generate a 150-channel binary semantic segmentation mask for each image. In Figure 2 , we visualize the segmentation masks in a color palette (rather than in 150-channel). The model trained on this mask achieves 67.6% accuracy, well above the chance level. Object detection provides coarse spatial annotations for objects through rectangle bounding boxes. We use ViTDet-Huge [35] trained on LVIS [24] with 1203 object categories to derive bounding boxes from each image. We add object class names to the bounding boxes to account for semantic meaning. This representation reaches 61.9% dataset classification accuracy, below semantic segmentation.Image captioning discards all visual information and produces semantic representations through natural language. Captions are less affected by pixel variations and spatial cues in images. Using LLaVA 1.5 [38, 37] , we generate a single-sentence caption and a long paragraph caption for each image. Short captions are shown in Figure 2 , and long captions are in Appendix D. We finetune a sentence embedding model Sentence T5-base [49] to perform dataset classification on these captions. This results in 63.8% accuracy with short captions and 66.1% with long ones, both nearing the accuracy for semantic segmentation. The richer details in longer captions enhance performance.Variation autoencoder (VAE) [32] encodes each image into a latent vector and then reconstructs the original image from it. The datasets may use different JPEG compressions or image resolutions, which could be exploited as shortcuts by dataset classification models. However, VAE's low-dimensional latent space may encode semantic information and suppress such low-level signatures. Reconstructing the images with a pretrained VAE from Stable Diffusion [57] only slightly decreases the accuracy from 82.0% to 77.4%, suggesting that low-level bias may have a smaller impact than semantic bias.Semantic segmentation, object detection, and caption extract semantic information with decreasing levels of spatial information. On the other hand, VAE could potentially reduce low-level signatures while preserving the semantics. The high accuracies of dataset classification models indicate that semantic bias is an important component of dataset bias in YCD.Next, we analyze the dataset bias rooted in object shape and spatial geometry rather than object semantics. To capture such structural visual bias, we use the Canny edge detector and the Segment Anything Model (SAM) [33] to outline object contours, and the Depth-Anything-V2 model [74] to measure pixel-level depth. While contour delineates fine-grained object shape, and depth estimation offers relative object positions, both lack the rich object semantic details present in semantic segmentation masks and bounding boxes (e.g., object class). Figure 3 visualizes the transformations. Segment Anything Model (SAM) [33] can provide high-quality object segmentation masks. We could then use them to delineate cleaner and more accurate shapes of objects that are minimally affected by local pixel variations, compared to the Canny edge detector. Specifically, we use SAM with the ViT-Large backbone to generate class-agnostic segmentation masks and identify boundaries by finding pixels whose surrounding pixels are not all from the same object. The accuracy on SAM contours (73.2%) is slightly higher than the one using the Canny edge detector (71.0%).Depth estimation captures the scene's spatial geometry, offering fine-grained spatial context and relative object positioning. Like contours, it excludes explicit semantic information about objects. The Depth-Anything-V2 (ViT-L) model [74] generates pixel-level depth estimation, encoded as a normalized grayscale image. The resulting 73.1% accuracy is comparable to that of SAM contours.The dataset classification accuracies on object contours and depth are even higher than the ones on semantics. This shows that object shape and spatial geometry variations are significant among YCD.To further understand the level of bias captured in spatial information as opposed to semantics, we keep the RGB values of all pixels in each image unchanged but shuffle the pixel positions to disrupt spatial coherence. We shuffle each image on the pixel level and the patch level, following a fixed order and a random order for all images. Figure 4 shows the images shuffled in a random order. Pixel shuffling obfuscates the image classifier with a permutation of the pixels and forces it to find patterns from the color distribution of pixels in each image. As expected, this significantly decreases the accuracy to 52.2% for the random order and 58.5% for the fixed order.Patch shuffling first divides each image into smaller patches and then rearranges the order of patches. Consequently, it preserves more local spatial information. Here we vary the patch size and show the results in Figure 5 . We note that the performances of the fixed order and the random order are almost identical with a patch size larger than 1. Surprisingly, with a patch size of 16, we reach almost the same accuracy as the reference one.The significant performance drop with pixel shuffling shows completely destructing the local structure in YCD can reduce its dataset bias to a large extent. However, the minimal accuracy decrease after shuffling patches of size 16 indicates patch-level local structures in spatial information is sufficient for identifying visual signatures of the YCD datasets.The high classification accuracy after pixel shuffling implies a discrepancy in pixel color distributions among YCD. To further assess this difference in color statistics among datasets, we transform each image into its average value for each color channel. Figure 6 shows the resulting images. In Figure 7 , we visualize the distribution of mean RGB values for YCD. There is only a moderate difference in mean RGB distribution between CC and DataComp. However, YFCC is much darker than CC and DataComp. This is further suggested by the confusion matrix of the dataset classification model trained on mean RGB images shown in Figure 7 , where the model classifies YFCC accurately but shows more confusion when distinguishing between CC and DataComp.Neural networks tend to exploit texture patterns even when the recognition task is inherently about the semantics [31, 20, 19 ]. If we decompose visual signals by frequencies, high-frequency bands typically capture these texture patterns and sharp transitions, whereas low-frequency components represent general structure and smooth variations. To explore how different frequency components contribute to dataset bias, we apply high-pass and low-pass filters to the original images.High-pass filter and low-pass filter. To filter signals based on frequencies, we first perform a 2D Fast Fourier Transform on each grayscaled image to obtain its representation in the frequency domain.We then apply an ideal filter [21] with a hard threshold radius of 40 in the frequency domain, so as to only keep either high (i.e., high-pass filter) or low (i.e., low-pass filter) frequencies. The filtered results are finally inversely transformed to the original grayscale domain, visualized in Figure 8 .Additional results and visualizations are in Appendix B.6. The model trained on images with high frequencies kept has an accuracy of 79.2%. This is slightly better than the one trained on images with low frequencies kept, which has an accuracy of 70.4%. Both accuracies are close to the reference accuracy of 82.0%.The high accuracy of models trained on either frequency component indicates that visual bias in the YCD datasets exists in both low-frequency and high-frequency components.Diffusion models [55, 45, 57] can generate synthetic images from a learned distribution. Synthetic images hold significant potential in augmenting data for a variety of vision tasks [26, 3, 65, 1, 5] . However, if dataset bias is inherited in the generation process, bias may be amplified as generated images are added to the training set. To understand this, we run dataset classification on synthetic images generated from unconditional and text-to-image diffusion models, illustrated in Figure 9 . Text-to-Image generation potentially preserves the semantic bias in the original images. We generate synthetic images from the popular SDXL-Turbo [58] diffusion model, condition on short captions produced by LLaVA (Section 3.2). Note unlike the unconditional generation above, we did not train our own text-to-image model for each dataset. By mapping the original images into an intermediate text modality and then back to the visual domain, we only retain the semantics captured in captions. We reach 58.1% accuracy, falling slightly short of 63.8% when directly classifying short captions.The high classification accuracy from unconditionally generated images shows that bias can be inherited in the synthetic images. The ability to classify text-conditional synthetic images further confirms that semantic discrepancy is a major contributor to dataset bias. Summary. We analyzed the impact of various transformations on dataset classification, identifying semantics and structures as important contributors to dataset bias. Patch-level local structure information is sufficient to classify YCD, with datasets differing even in color statistics. Bias spans across frequency components, particularly in high-frequency bands. Finally, we showed that bias can be inherited in synthetic images of diffusion models. More results are in Appendix B.Descriptions focus on colorful and lively scenes, with vibrant attire, festive settings, and active engagements, emphasizing visual appeal and movement.Descriptions often emphasize close-up shots, highlighting the intricate details, textures, and designs of objects, with a focus on aesthetic and functional attributes.Many images feature serene outdoor environments, including natural landscapes, gardens, and bodies of water, highlighting a calm and peaceful atmosphere.Descriptions focus on colorful and lively scenes, with vibrant attire, festive settings, and active engagements, emphasizing visual appeal and movement.Descriptions often emphasize close-up shots, highlighting the intricate details, textures, and designs of objects, with a focus on aesthetic and functional attributes.Many images feature serene outdoor environments, including natural landscapes, gardens, and bodies of water, highlighting a calm and peaceful atmosphere.Often provides detailed descriptions of attire, personal items, and their positioning within the image.Product images commonly feature visible brand names, logos, or technical specifications.Often portrays dynamic scenes with people or animals in motion, emphasizing the energy and liveliness of the captured moments.The images often have a more stylized or artistic quality, with a focus on composition, lighting, and texture, and may include text, logos, or other visual elements that add context or meaning to the scene.The images often feature objects, products, or scenes that are more static or still, such as a product on a white background, a diagram or chart, or a scenic view, with a focus on showcasing their design, texture, and visual appeal.The images often feature vibrant colors, dynamic compositions, and a sense of energy or movement, capturing the beauty and excitement of the world around us. 
explaining semantic bias among datasets: 
 In the preceding section, we explore transformations to extract various types of image information, each exhibiting varying levels of bias. Among them, semantic bias heavily contributes to the high accuracy in the dataset classification task [39] . In this section, we identify specific interpretable semantic patterns within each dataset through object-level and language-based analysis. Grad-CAM [80, 60] highlights key regions in an input image that explain neural network predictions.In Figure 10 , applying Grad-CAM to the reference model (Section 3.1) shows that the model focuses on semantically meaningful objects: elephant herd in the third image, table and chair in the fourth image, and pen in the sixth image. This suggests that the model might have leveraged the object-level information to recognize the dataset identity of each image. To better understand this, we apply models pretrained on other vision datasets (ImageNet-1K [15] , LVIS [24] , and ADE20K [81] ) to provide object annotations for each image in the datasets, and analyze their object-level bias. As a result, the analysis below is in the context of 3 sets of object classes, defined by these 3 datasets.Imbalanced object distribution. Imbalance in object distribution is a common form of semantic bias.For each object class, we calculate the number of images in each dataset that contain the object class and their percentage share relative to all images with that object class. Note for LVIS and ADE20K models' output, we counted each object class only once per image, even if multiple instances or pixels of the same object class are present. Figure 11 shows the top 8 object classes with the highest percentage of images from YFCC, CC, or DataComp. The dominance of a certain dataset within these classes highlights a considerable imbalance in object-level distribution across datasets. Figure 11 also shows that YFCC constitutes much higher proportions in its top object classes than CC and DataComp in their respective classes (note the different x-axis scales in each subplot). To further see this, we visualize the distribution of unique object class counts per image in Figure 12 . The higher variety of objects in YFCC images shows a notable gap in object diversity among YCD. On average, YFCC contains the highest number of unique objects in each image, followed by CC, while DataComp exhibits the lowest.The coefficients of a logistic regression model form a natural importance ranking of input features when the features are binary. To leverage this, we represent each image with a binary vector, where each element indicates the presence of a specific object class from a set of objects (i.e., ImageNet, LVIS, or ADE20K). We train a logistic regression model to predict the dataset origin of the images based on their binary vector representations. This simple model achieves validation accuracies of 52.0% with ImageNet objects, 52.4% with LVIS objects, and 52.4% with ADE20K objects.Figure 13 shows the top objects based on logistic regression coefficients. It highlights outdoor infrastructures (e.g., traffic light, clock tower, telephone pole, and building) in YFCC and household These rankings partially overlap with the list of objects with the highest proportions among the YCD datasets (Figure 11 ). However, the object rankings also identify objects that are more balanced across datasets, since logistic regression receives more weight updates based on the more common objects during training. Thus, it provides a complementary angle on object-level bias among the datasets.The word clouds [47] in Figure 14 offer a visual representation of the most prevalent phrases in the long paragraph captions of each dataset from Section 3.2. We observe several frequent phrases featuring human subjects (e.g., people, group, wearing) in YFCC, elements of indoor scenes (e.g., room, and dining table) in CC, and a focus on white background in DataComp. In this section, we will use the long captions as a proxy to dive deeper into the semantic themes in each dataset. Unsupervised topic discovery. We treat each caption as a document and apply the Latent Dirichlet Allocation (LDA) [7] for topic discovery to each dataset in YCD (with the number of topics set to 5). Figure 15 presents the top 5 words for each topic. Notably, in YFCC, three topics (first, second, and fifth) feature words associated with outdoor scenes; in CC and DataComp, their topics cover "logo" and "design," suggesting the presence of digital graphics. This powerful ability of LLMs allows them to provide open-ended and detailed natural language explanations. Specifically, we procedurally prompt GPT-4o to extract dataset-specific characteristics from caption datasets and refine its answers into 5 bullet points, shown in Figure 16 . In summary, YFCC is characterized by abundant outdoor, natural, and human-related scenes, while DataComp concentrates on static objects and synthetic images with clean backgrounds and minimal human presence. In contrast, CC blends elements of both YFCC's dynamic scenes and DataComp's static imagery. Appendix A.2 provides the entire prompt structure and the complete LLM summarization output. We further verify the validity of the semantic features from LDA and LLM in Appendix C.Captions highlight specific objects and items of clothing, such as jewelry and accessories, with intricate details on patterns, colors, and designs.Many images showcase static, picturesque environments or artistic designs, creating a calm and detailed visual representation of both natural and structured settings.Descriptions typically include detailed elements of the scene, focusing on settings like trees, buildings, and objects in natural or urban environments, emphasizing the rich context and interactions between elements.Covers diverse environments, from intricate indoor layouts to historical scenes and unique scenarios.Descriptions highlight specific features, materials, colors, and designs of objects.Regularly includes flora and fauna elements, highlighting the presence of animals and natural features within the images.The images frequently focus on objects, concepts, or ideas, highlighting their significance, functionality, or emotional resonance, often with a mix of objects, people, and text.The images frequently include detailed, close-up views of objects or visual elements, such as intricate designs, patterns, or textures, which suggests a focus on craftsmanship and artistry.The images frequently include people as the main subject, with a focus on their actions, emotions, or interactions with their environment, often conveying a sense of energy, movement, or excitement. 
emotions and interactions ⋯: 
 Figure 16 : LLM summarization of dataset features. The bullet points highlight outdoor, natural, and human scenes in YFCC and static objects and synthetic images in DataComp. CC contains both YFCC's dynamic scenes and DataComp's static images.Summary. We leveraged closed-set object-level queries and open-ended language analysis to interpret the semantic bias among datasets. The object-based analysis identified objects indicative of each dataset within a predefined object set. On the other hand, natural language methods are able to provide open-ended explanations and rich details explaining the characteristics of each dataset.YCD have different sources: YFCC is selected with minimal filtering from user-uploaded images on Flickr.com, while CC and DataComp filter web-sourced images with high quality in caption, image, or their alignment. We build on the interpretable semantic bias in Section 4 to discuss the dataset curation procedures, and provide potential suggestions.Filtering based on a reference dataset or model may inherit its bias. DataComp has the fewest unique objects per image (Figure 12 ). This is possibly because DataComp filters for images with visual content close to ImageNet data in the embedding space [18] . Therefore, the remaining images tend to be object-centric [4] . It also filters for images that align well with its captions in CLIP [53] embedding space, therefore favoring certain types of images, e.g., images containing text. To mitigate this, we may use datasets or models that contain less bias themselves for filtering.The source website's image collection mechanism can introduce bias. We note that YFCC is heavily skewed towards outdoor scenes and human interactions (Section 4.2). This bias likely stems from its reliance on a single data source, Flickr.com, where user-uploaded content often focuses on personal photos, landscapes, and social interactions.Web images naturally contain more digital graphics. Since CC and DataComp images are from Internet webpages, professionally created content like advertisements, infographics, and digital media are prioritized. Dataset users should evaluate if this composition aligns with the downstream goals.There is a notable presence of artistic scenes including paintings, murals, sculptures, and elements of performance and fantasy, showcasing creativity.Scenes are often set against simplistic, plain backgrounds, drawing attention to the main subject and creating a clean, uncluttered look.Scenes frequently capture moments of celebration, competition, or personal interaction, highlighting the emotions and dynamic aspects of the subjects in various social and communal settings. Histograms of Oriented Gradients (HOG) [14] robustly represents gradient patterns by aggregating gradient information within small spatial regions (cells) into histograms. These histograms are normalized over a block of adjacent cells to improve invariance to changes in illumination and contrast. HOG visualization shows the dominant gradient direction in each cell, with brightness indicating gradient magnitude. The near-reference accuracy of 79.0% suggests that the shape and contour features captured by HOG are highly indicative of the dataset identity.Commonly features written elements, references to specific brands, characters, or pop culture icons.Imagery includes diverse elements like household items, abstract designs, and informational content such as diagrams or book covers.Provides rich descriptions of clothing, surroundings, and contextual elements, offering a comprehensive view of the scenes depicted. The images often feature objects, food, or other visual elements arranged in a visually appealing manner, with a focus on color, texture, or composition.The images often feature a focus on simple colors, clean backgrounds, and a focus on the main subject or information being conveyed, resulting in a sense of simplicity and minimalism.The images often feature everyday scenes, presented in a realistic and detailed manner, showcasing natural environments and ordinary activities.  
conclusion: 
 We proposed a framework to study the bias in large-scale visual datasets and used it to analyze three representative datasets. By classifying transformed images' dataset origin, we identified structures and semantics as key factors in dataset bias. We further investigated specific forms of semantic bias among datasets through fixed object queries, highlighting distinctive concepts characterizing each dataset. Lastly, we extracted key topics and natural language summaries for each dataset. We hope this framework and these findings can encourage further exploration of dataset bias and help improve diversity and representation in future datasets. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide the details about compute resources here. We use 8 NVIDIA 2080 Ti to train the ConvNeXt-T model for the dataset classification task with 8 gradient accumulation steps. The average compute time for each experiment is about 1.5 days.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We have followed the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited the papers for all datasets we leverage in the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Dataset Classification with Transformations Dataset classification training. Following Liu and He [39] , for image-text datasets CC and Data-Comp, we only use their images. For each dataset in the YCD combination, we uniformly sample 1M / 10K to form the train / val sets. To speed up image loading, the shorter side of each image is resized to 500 pixels if the original shorter side is larger than this, with the aspect ratio preserved. We observe this has minimal effect on model performance. Data processing. By default, our training setting uses RandomResizedCrop and RandAug [13] as data augmentations. Note for transformations involving pre-trained models, transforming every augmented image can be time-consuming. To address this, we apply data augmentation after these transformations. This ensures that the transformed image remains consistent throughout training. In addition, RandAug is not used for binary or grayscale inputs due to its incompatibility. Table 2 shows whether we apply data augmentation after transformation and whether we use RandAug. Other details. In Section 3.2, for classifying captions, we finetune the Sentence T5-base [49] model on generated captions using a fixed batch size of 128. To determine the optimal configuration for learning rate and training epochs, we conduct a 2D parameter sweep, selecting the learning rate from {1e-3, 1e-4, 1e-5} and training epochs from {1, 2, 4, 6}. The number of warmup iterations is set to epochs×1562 iterations. For VAE, we use the model from [57] with a downsampling factor of 4, which encodes an RGB image of shape 256×256×3 into a latent vector of size 64×64×3. In Section 3.7, we train a DiT-B/2 [51] model on each of YCD datasets for 275K iterations, using a batch size of 1024 and a fixed learning rate of 1e-4.Here are some samples from three different distributions of captions. We access GPT-4o through ChatGPT Temporary Chat in May 2024. Our in-context learning prompt structure is shown in Figure 17 . For each iteration, we sample 120 long captions (Section 3.2) from each of the YCD datasets, creating a shuffled set of 360 in-context examples. To prevent LLM from relying on prior knowledge about any of the YCD datasets, we randomly assign an index from {1, 2, 3} as the name for each dataset in each iteration. We then prompt the LLM to determine the source distribution of the validation sample. Inference is performed on one validation caption at a time.Figure 18 shows the validation accuracy as a function of the number of validation samples per class. We stop sampling more validation captions at 160 samples per class, where the accuracy stabilizes.We use a two-step pipeline to prompt GPT-4o to extract semantic features based on 10 batches of captions, where each batch consists of 120 randomly sampled captions from each dataset. The model refines its responses based on the aggregated list of extracted features. Figure 19 illustrates the pipeline and specific prompts used. Figure 20 shows the full LLM summarization output. We further explore how the performance of the pretrained models used for transformations in Section 3 affects our results. Specifically, we perform object detection, contour extraction, and depth estimation using smaller and less powerful models ViTDet-Base, SAM (ViT-Base), and Depth-Anything-V2 (ViT-B). As shown in Table 3 , the pretrained model size minimally impacts the classification accuracy. Additionally, we apply our LLM summarization method in Section 4.2 to different LLMs: Claude 3.5-Sonnet [2] and Llama-3.1-8B-Instruct [16] . Figures 22 and 23 list the 5 summarized dataset features. The features for each dataset from different LLMs express highly similar concepts.In Section 3, we employ ConvNeXt-Tiny as the single base classifier to perform the dataset classification tasks on transformed images. To see whether our results are still valid for models with different sizes and architectures, we further train ConvNeXt-Femto [72] , ConvNeXt-Nano, and ResNet-34 [25] to perform the dataset classification tasks in Section 3. In Table 4 , we report the validation accuracies on the YCD datasets applied with various image transformations. [9] with a threshold of 30.We also finetune two additional sentence embedding models MPNet-Base [62] and Sentence-BERT-Base [56] for dataset classification on YCD captions generated from LLaVA. Both of these models are weaker [48] than our default Sentence-T5-Base in Section 3.2. Nevertheless, as shown in Table 5 , the dataset classification accuracy remains consistent across models and transformations. We are interested in whether combining several visual attributes can jointly contribute to a larger dataset bias. To this end, we perform dataset classification on the concatenation of two copies of the same original images applied with two different transformations. We consider all pairwise combinations of object detection, pixel shuffling, and SAM contour. Table 6 shows the resulting classification accuracies. We observe that combining semantic and structural attributes can result in higher dataset classification accuracy compared to using a single attribute alone.To show that the high validation accuracy on the transformed datasets is achieved through generalization rather than memorization of training examples, we follow Liu and He [39] to perform the classification of pseudo-datasets after applying transformations.Specifically, we create three pseudo-datasets, each sampled without replacement from the same transformed YFCC dataset. Tables 7 and 8 We further investigate how varying thresholds affect dataset classification performance on low-pass and high-pass filtered images. Figures 24 and 25 display transformed images with ideal filter [21] and Butterworth filter [9] . Figure 26 presents the resulting dataset classification accuracies across different thresholds. While accuracies generally decline at extreme threshold values, in other cases, ideal and Butterworth filters achieve high accuracies for both low-pass and high-pass filters. It is worth noting that the Butterworth filter employs a soft thresholding approach when filtering frequencies, therefore allowing each image to retain some frequency information beyond the selected threshold. This likely contributes to the high accuracy observed on high-pass filtered images, even with a higher threshold setting. Additionally, we observe that the high-pass ideal filter can achieve a higher classification accuracy with a threshold of 150 compared to 115. This is likely because when the threshold is at 115, the biased higher-frequency information is dominated by less-biased lower-frequency information. [17] . Each cell lists the top 3 concepts distinguishing the dataset in each column from the one in each row. Note the VisDiff concepts highly overlap with the dataset characteristics from our language analysis in Section 4.2.VisDiff [17] is a recently proposed method to describe differences between two image sets in natural language. It first uses a VLM to generate image captions and then leverages an LLM to propose concepts that distinguish the first image set from the second image set based on these captions. For each concept, VisDiff uses the CLIP similarity scores between the images and that concept to classify two image sets, and calculates the AUROC of this classification. The final output concepts are ranked based on their corresponding AUROC scores.For each dataset pair in the YCD datasets, we use VisDiff to identify the top 5 concepts that better describe the first dataset compared to the second one, as shown in Figure 28 . VisDiff's concepts closely align with dataset characteristics of our language analysis in Section 4.2, highlighting "people" and "outdoor activities" for YFCC and "products" for DataComp. However, our method can also identify specific visual biases across datasets with various transformations, fixed object-level queries, and in-depth natural language analysis.Table 10 shows examples of long paragraph captions generated by LLaVA 1.5 [37] in Section 3.2. dataset images long paragraph caption YFCC A black and white dog is standing in a shallow body of water, possibly a river or stream. The dog appears to be drinking from the stream, surrounded by rocks. The scene captures the dog's natural behavior as it quenches its thirst at a water source.A food truck is parked on the side of the road, with a man standing nearby. The truck is blue and features a surfboard on the side, indicating a surf theme for its offerings. The man, possibly the owner or an employee, stands beside the truck, either waiting for customers or preparing to serve food.A large herd of elephants is walking across a grassy field, with at least 13 elephants visible. Some are closer to the foreground, while others are further in the background, spread out across the field in pairs or individually. The scene captures the majesty of these creatures in their natural habitat.A cozy bedroom featuring a bed, desk, and chair. Above the desk, a TV is mounted on the wall, and a laptop rests on the desk, accompanied by a vase of flowers that adds elegance to the room. The space also includes a dining table and chairs, providing versatility for different activities.A spacious dining room with a large wooden table surrounded by chairs. The table is set with a white tablecloth, and pillows on the chairs add to the cozy ambiance. Natural light fills the room, and potted plants are placed around the area, enhancing the overall atmosphere.A notebook and pen are placed on a table next to a lollipop. The notebook, open with a polka dot design, lies beside the pen, ready for use. The colorful lollipop adds a playful element to the scene.Table 10 : Examples of long paragraph captions generated from LLaVA 1.5 [37] . The difference in object distribution among datasets using ImageNet object queries in Section 4.1 can be further used to perform dataset classification. Specifically, we classify an image into the dataset that has the highest frequency of this image's label. Without any learnable parameters (in contrast to logistic regression in Section 4.1), this simple decision rule achieves a validation accuracy of 50.41%.The model trained on original datasets in Section 3.1 might also use the imbalance of underlying object-level distributions. To demonstrate this, we partition both the original YCD datasets and the accuracy of the dataset classification model trained on those by the ImageNet object annotations in Section 4.1. Figure 29 shows the accuracy for each object class vs. its majority dataset share. For each class in the partition, the greater the majority dataset share among the YCD datasets is, the higher the corresponding accuracy of the model is.Since our framework relies heavily on pretrained recognition and generative models [57, 12, 35, 33, 38] for extracting semantic and structural information from images, the analysis may be affected by the bias inherent to those models or the datasets they are trained on. For example, if the pretrained models are trained on data very similar to a certain dataset under study (e.g., one of YCD), the measured level of bias may be affected. In addition, dataset classification can only reveal bias by comparing multiple datasets, and cannot be directly applied to a single dataset to understand its bias.Our framework can be used to analyze the datasets before model training, to better determine whether the dataset composition aligns with training goals. It is relatively a fast process compared to a full training cycle. This can help researchers reduce experiment iterations and thus total energy usage.• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Note that in this work we study bias among datasets. This concerns the coverage of concepts and content (i.e., how representative the dataset is for the real world). This is related to but different from another notion of bias that concerns social and stereotypical bias as well as algorithm fairness[8,73,6,46]. 


Decision: All reviewers were positive about the paper after the rebuttal. The authors have done substantial new work (experiments, analyses, etc.) in the rebuttal stage, which helped convince the reviewers. AC suggests the authors incorporate them into the revised paper.

Reviews: review: paper_summary: paper_summary: This paper studies the problem of dataset bias prevalent in current multimodal datasets. It revisits the dataset classification experiment from Torralba et al, recently studied again by Liu and He, and deconstructs their findings to understand what aspects of datasets (structural, semantic, color, object-level, caption-level etc) contribute most to the bias prevalent in visual datasets. With abundant empirical evidence, the paper provides an interesting and important analysis on the fundamental gaps in our understanding of large-scale image-text datasets.
strength_weakness: strength_weakness: I have a few technical concerns that might undermine the significance of this paper's findings, I note them down here. Further, I also have a few suggested additional experiments that might help boost the significance of this paper's results.

- All the results rely on a three class classification problem at heart. How should one interpret these results under the argument that neural networks in general can learn noisy labels, and can fit arbitrary distributions [1]? How does the baseline 81.7% performance change if you assigned random labels to the original set of 3M images? This is a very important ablation to verify the significance of all the findings.

- One issue with the semantic segmentation classification experiment (sec 3.2) is that it removes the pixel information from the task, and converts it into a potentially much simpler task where the input features are only of size 150 (if I understand the experiment correctly). Here is a simple suggestion for removing that confounder. What happens if you train directly on the semantic masks itself? Of-course there would be some bias coming in from the colour applied to the segmentation masks for each of the different object classes, but this could be mitigated by training two different models using two different mask colour palettes, and looking at the variance. If the accuracy of this particular task still remains similarly high, that would further boost the significance of the results.

- For the caption classification task (sec 3.2), could you also use another sentence embedding model that is potentially weaker in its initial MTEB [2] performance, for reproducing the results of that experiment. Again, this would help boost the significance of the findings. 

- Would the main results be consistent across two completely different subsets of the YCD classificaiton task? Could you sample an entirely different set of 3M images and 30K validation images, rerun the experiments, and cross-validate on the two different validation sets?

- For figure 11, could you redo that analysis on another subset of 3M samples as well? I am skeptical that this result would hold exactly true for a given dataset, especially given this prior result that most datasets curated from the web have similar concept distributions [3]. Can the authors provide a reconciliation between this prior result and their results?

- Another interesting analysis that can be done would be to use the VisDiff tool [4] to perform an analysis on the visual distribution differences between the pairs of YCD datasets. It might provide an interesting sub-analysis and would help further verify the results in sec 4.2 and fig. 16 in the paper.

- The paper has a lot of very interesting insights about bias across visual datasets. However, there are no concrete suggestions/discussion on how one could go about mitigating these biases. A discussion regarding the source of these datasets, different data curation / filtering mechanisms, and how they might potentially impact these biases, would make for a great and required addition to this paper.

[1] Zhang et al, Understanding deep learning requires rethinking generalization
[2] Muenighoff et al, MTEB: Massive Text Embedding Benchmark
[3] Udandarao et al, No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
[4] Dunlap et al, Describing Differences in Image Sets with Natural Language
questions: questions: All my questions are also mentioned in the weaknesses section above.
limitations: limitations: Yes the authors have mentioned that their main weakness is with respect to using pretrained models which themselves might be biased.
paper_summary: paper_summary: This work theorizes and investigates various concrete forms of inter-dataset biases, namely those among YFCC, CC, and DataComp (YCD). The authors analyze such inter-dataset biases in pure visual attributes as well as in semantic attributes using LLM-generated descriptions.
strength_weakness: strength_weakness: This work extends the scope of [1] and aims to provide specific insights for constructing less biased data collection in future. However, I feel there are several missing evidence that undermine the contributions of this work in its current state:

1. **The authors have only examined the factors of biases individually.** However, I notice that none of the single attributes listed contributes to a higher prediction accuracy over the baseline of using original visual features. Could it be possible that a combination of multiple visual/semantic attributes jointly contributes to the large overall bias? This needs to be verified.

2. **The authors have only investigated one classification model.** According to [1], the inter-dataset overall bias (high prediction accuracy) is observed with multiple classification models as well. So if we use the smaller ResNet-50 model, can we still observe the high accuracy over each individual structural or semantic attribute? I haven't seen the authors ruling out the confounding factor of the classification model size.

3. **The proposed framework consists of indicative bias metrics only relative to YCD.** However, YCD all have their own intra-dataset biases. So when we are attempting to create a debiased dataset in future, *how can we make sure our newly collected data are truly diversified and fair?* Can the authors provide similar evidence over 'Memorization vs. Generalization' as in [1] Sec. 4.2., and verify individual inter-dataset bias attributes with a truly unbiased pseudo-dataset?

[1] A Decade’s Battle on Dataset Bias: Are We There Yet? Liu et al. 2024.
questions: questions: Please find my major questions in the Weakness section.

Another slight concern is about the presentation. Since an ideal debiased dataset should enjoy an equal likelihood to be predicted as one of the anchor datasets (e.g. YCD), maybe the authors should explicitly clarify this important task setting as early as possible in the paper.

(Aug 9th): Updating my overall ratings thanks to the quality response.
limitations: limitations: The limitations have been sufficiently addressed.
paper_summary: paper_summary: This paper explores dataset biases and introduces a framework to identify the unique visual attributes that differentiate various datasets. The method involves applying a range of transformations to extract semantic, structural, boundary, color, and frequency information from the datasets, evaluating how each type of information contributes to their distinct characteristics.
strength_weakness: strength_weakness: 1. Even though the paper identifies various factors that help in distinguishing datasets, it does not provide any information/ takeaways on how this information could be used to 'build more diverse and representative datasets in the future'.I consider the investigative analysis as a good and significant contribution but not commenting on how to utilise these results in a way that would be useful for the community is a significant shortcoming of this paper. The authors could have moved some of the transformations to the appendix and utilise the space for describing 'key takeaways and explaining how &what dataset curators should keep in mind while building datasets.
2. No information is provided on how the accuracy of pre-trained models affects the inferences or findings. 
3. There is no discussion on how the format of data transformation affects the findings. For instance, converting the dataset into segmentation masks or object detection boundaries could capture more information in the segmentation results. The authors have not addressed the impact of these different transformation formats on their observed findings.
questions: questions: 1. how does the accuracy of pre-trained models affect the inferences or findings?
limitations: limitations: Yes.

Comments: title: Thank you
Reply
Invitation to discussion
Thank you
Reply to "build more diverse and representative datasets in the future"
Response to rebuttal
Reply
Thank you
This is some impressive response.
Rebuttal
Rebuttal
Rebuttal
comment: Many thanks for your valuable feedback and discussion!
My concerns are mostly addressed, it would be interesting to see how these suggestions could be incorporated to build a new dataset curation method in future work. I increase my score to 6 after reading the rebuttal.
Dear Reviewer aLTB,

We would like to invite you for any further discussion, as the reviewer-author discussion stage is approaching the end (in 1 day). We hope our response can address your previous concern on how "our findings can be used to build more diverse and representative datasets in the future". We want to make sure we have the chance to clarify/respond again if you have any further concerns. Thank you!

Best,\
Authors
Thank you for your time and feedback on the paper! We’re glad that your concerns have been resolved. We will make sure to include the additional results from the rebuttal in the next version of our paper. The review has been very helpful in enhancing our paper.
Thank you for your clarifications. Indeed, our previous response focused more on past efforts. **Here we provide several ways our framework can be used to help build more diverse and representative datasets in the future:**

- When considering adding a new set of images (e.g., from another website) into a data collection, we can first treat new images as a separate dataset, use the transformation and classification framework to tell where and how much they differ from the existing image collections. This can help us decide whether to join them. If the goal is to enhance diversity of the dataset in a certain aspect (e.g., object types), we should only join them when they are sufficiently different.

- Our language-based analysis provides textual descriptions for any new dataset. This text description directly gives the data curators the intuition on the gist of the dataset, especially when compared with reference datasets. It can help curators refine text terms for image search and tag filtering, in search engines or other platforms.

- Our framework can identify bias and distribution imbalance in the image statistics (e.g., colors and object distribution). This can help guide adding/removing images with desired/undesired statistics for more balance.

- The result of a dataset classifier trained on the transformed datasets can serve as a measure of the image's "typicality" within the dataset. For example, images in dataset A that are misclassified as images in dataset B are considered "not typical" within A, for that transformation/attribute. If needed, images with less typical attributes can be then oversampled in collecting data and/or training to enhance representation.

**In addition to how to use the framework, we list a few direct lessons we learned from our analysis on YCD, that can also help build more diverse and representative datasets in the future:**

- Filtering by embedding similarity to images of a reference dataset could inherit bias of that dataset.

We observe that DataComp has the lowest number of unique objects per image (Fig 12). This potentially resulted from DataComp filtering for images with high embedding similarity to ImageNet training examples, most of which feature object-centric images [1].

To mitigate this, dataset curators should be mindful of the inherent biases in their reference datasets (e.g., ImageNet). Concretely, to enhance the object diversity within DataComp, one could consider using a reference dataset with a higher per-image object diversity (e.g., COCO) during filtering.

- The source website's image collection mechanism can introduce bias.

We also noted that YFCC is heavily skewed towards outdoor scenes and human interactions (Sec 4.2). This bias likely stems from its reliance on a single data source, Flickr, where user-generated content often focuses on personal photos, landscapes, and social interactions. Dataset curators should recognize that the collection methods (e.g., user uploads) of data sources (e.g., Flickr) can introduce biases into the resulting dataset (e.g., YFCC).

- Web-scraped images would naturally contain more digital graphics.

Since CC and DataComp are crawled from the Internet, they feature results from search engines [2, 3]. This prioritizes professionally created content like advertisements, infographics, and digital media. Curators should evaluate whether this composition aligns with their downstream goals.

Thank you for your quick reply/clarification. We will add a discussion on this to the paper and see it as an important improvement. We also hear your suggestion on moving experiments to the appendix for the space. Given that NeurIPS policy allows an additional page for accepted publications, we would be able to add the discussion while maintaining the current experiments. We are happy to address any further concerns.

References:\
[1] Barbu et al, ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models\
[2] Changpinyo et al, Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts\
[3] Gadre et al, DataComp: In search of the next generation of multimodal datasets
I thank the authors for their added experimental results and comments.

(1) The random labels experiment sufficiently answers my question regarding the specificity of these dataset-level biases.

(2) Thanks for running the sentence embeddings and segmentation mask experiments, those high stable accuracies clarify my concern.

(3) The cross-validation results are very strong. I would encourage the authors to somehow incorporate these results into the main paper perhaps as a mean +/- std or something similar, I think this would significantly increase the confidence in the paper's findings.

(4) Thanks for the clarifying discussion b/w your work and the No Zero-shot paper. You being able to reproduce the findings of that paper are very interesting.

(5) Great that the VisDiff results corroborate your analysis.

(6) Your response to w7 is very insightful, thank you for adding this. I would encourage to expand on this discussion and if possible add some sample images from each of the datasets to that section in the appendix. Would be super useful!

Overall, the author rebuttal has significantly increased my confidence in the paper and the main results and insights provided in the paper are very useful. I am updating my score to an 8.
Thanks for the feedback.

I think the authors misunderstood my initial review, I didn't mean how "the identified dataset bias can be used retrospectively to analyze the dataset curation procedure". I asked **how these findings can be used to  'build more diverse and representative datasets in the future'?** .
Thank you again for your helpful comments and for reviewing our response! We are glad to hear that the concerns have been addressed. We will incorporate the additional results discussed in our rebuttal into the next version of the paper and emphasize that we're studying the inter-dataset bias in our introduction to better reflect the focus of our work.
I greatly appreciate the authors effort in putting up the response. In fact, it's beyond my expectation. I find my 3 major concerns have been adequately addressed thanks to the additional information - I trust the authors will incorporate them into the final version.


In retrospect, I believe what hindered my initial impression was the title of the paper, which led me to believe this was some incremental work over investigating *'intra-dataset biases'*, such as imbalanced distribution of object classes or visual features and etc. I highly recommend, if applicable, that the authors explicitly open up with **'INTER-dataset biases'** in the title or somewhere early-on in the revised paper. Overall, I am willing to update my rating owning to this thorough rebuttal.
We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] Thomee et al, YFCC100M: The New Data in Multimedia Research\
[2] Gadre et al, DataComp: In search of the next generation of multimodal datasets\
[3] Changpinyo et al, Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts\
[4] Li et al, Exploring Plain Vision Transformer Backbones for Object Detection\
[5] Kirillov et al, Segment Anything\
[6] Podell et al, SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis\
[7] Anthropic, Claude 3.5-Sonnet\
[8] Meta, Llama 3.1
We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] Zhang et al, Understanding deep learning requires rethinking generalization\
[2] Song et al, MPNet: Masked and Permuted Pre-training for Language Understanding\
[3] Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks\
[4] Udandarao et al, No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance  \
[5] Dunlap et al, Describing Differences in Image Sets with Natural Language\
[6] Thomee et al, YFCC100M: The New Data in Multimedia Research\
[7] Gadre et al, DataComp: In search of the next generation of multimodal datasets\
[8] Changpinyo et al, Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] He et al, Deep Residual Learning for Image Recognition\
[2] Liu et al, A ConvNet for the 2020s\
[3] Song et al, MPNet: Masked and Permuted Pre-training for Language Understanding\
[4] Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks\
[5] Thomee et al, YFCC100M: The New Data in Multimedia Research\
[6] Gadre et al, DataComp: In search of the next generation of multimodal datasets\
[7] Changpinyo et al, Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts\
[8] Liu et al, A Decade’s Battle on Dataset Bias: Are We There Yet?